[Swift-user] Channel Timeout on Beagle?

Michael Wilde wilde at anl.gov
Fri May 29 11:41:00 CDT 2015 

Matthew,

You should consider using Swift 0.96.0, and to the extent possible use 
local filesystems instead of the shared filesystem, which is often under 
excessive load.

We can discuss how to do this in subsequent followup as needed. 
Basically, try provider-staging, and put both the input data on the 
login node's local filesystem, and the site workdirectory under /dev/shm 
or /tmp. (You may need to probe the compute node as to which of these is 
writable and has sufficient space).

- Mike

On 5/29/15 10:39 AM, Matthew Shaxted wrote:
>
> It looks like the timeout problem is not solved actually. For some 
> reason I am having much difficulty running on Beagle, and I have an 
> feeling it is due to slow read/write.
>
> For example, I finished ~1,200 / 12,000 runs before failure (see below 
> paragraph) and moving these results (of not very large result files) 
> to the public_html is taking an hour or so. I’m hoping to scale up to 
> 100-300k runs or so, thus this will become a significant bottleneck. I 
> have emailed beagle-support about this issue just now.
>
> In all test environments my Swift workflow is working well, but when 
> submitting jobs to Beagle queue, it completes some number of 
> simulations before the timeout error occurs and all jobs stop. I'm 
> using Swift-0.95-RC7 (and am in process of updating to 0.95 latest), 
> but think these errors may also be due to this slow read/write.
>
> Any suggestions?
>
> Below is the error I see and the job completely stops:
>
> Host: cluster
>
> Directory: epsweep-run004/jobs/a/RunEP-ai2mic9m  exception @ 
> swift-int-staging.k, line: 181
>
> Caused by: exception @ swift-int-staging.k, line: 177
>
> Caused by: Block task failed: Connection to worker lost
>
> org.globus.cog.coaster.TimeoutException: Channel timed out. 
> lastTime=150526-142313.128,
>
> 50526-142514.107, channel=TCPChannel [type: server, contact: 
> 0526-0802460-000014-000456
>
> at 
> org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
>
> at 
> org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
>
> at java.util.TimerThread.mainLoop(Timer.java:566)
>
> at java.util.TimerThread.run(Timer.java:516)
>
> MATTHEW SHAXTED
>
> SKIDMORE, OWINGS & MERRILL LLP
> 224 SOUTH MICHIGAN AVENUE
> CHICAGO, IL 60604
> T  (312) 360-4368
> MATTHEW.SHAXTED at SOM.COM <mailto:MATTHEW.SHAXTED at SOM.COM>
>
> cid:image001.png at 01CF9071.6FB46030 <http://www.som.com/>
>
> The information contained in this communication may be confidential, 
> is intended only for the use of the recipient(s) named above, and may 
> be legally privileged. If the reader of this message is not the 
> intended recipient, you are hereby notified that any dissemination, 
> distribution, or copying of this communication, or any of its 
> contents, is strictly prohibited and may be unlawful. If you have 
> received this communication in error, please return it to the 
> sen­­­der immediately and delete the original message and any copy of 
> it from your computer system. If you have any questions concerning 
> this message, please contact the sender.
>
> http://intranet.som.com/common/admin/file.cfm?f=%2Fresources%2Fcontent%2F5%2F0%2F4%2F4%2F6%2F4%2F0%2F3%2Fdocuments%2Fimagea560bf%2Egif%406e10073b%2E30854c37
>
> *From:* Matthew Shaxted
> *Sent:* Wednesday, May 27, 2015 2:04 PM
> *To:* 'Swift User'
> *Subject:* RE: Channel Timeout on Beagle?
>
> Hi All: I was able to get the runs working successfully by changing 
> the maxtime flag in the sites file.
>
> Thanks
>
> *From:* Matthew Shaxted
> *Sent:* Wednesday, May 27, 2015 9:50 AM
> *To:* Swift User
> *Subject:* Channel Timeout on Beagle?
>
> Hi Swift Users:
>
> I am running some studies on Beagle using Swift, and experiencing a 
> strange error. The Swift scripts run great on cloud and on the Beagle 
> login node, but seems to be timing out for some reason.
>
> Does anyone have insight into the cause of this? Thanks for any help.
>
> Below is the error I am getting:
>
> Host: cluster
>
> Directory: epsweep-run004/jobs/a/RunEP-ai2mic9m  exception @ 
> swift-int-staging.k, line: 181
>
> Caused by: exception @ swift-int-staging.k, line: 177
>
> Caused by: Block task failed: Connection to worker lost
>
> org.globus.cog.coaster.TimeoutException: Channel timed out. 
> lastTime=150526-142313.128,
>
> 50526-142514.107, channel=TCPChannel [type: server, contact: 
> 0526-0802460-000014-000456
>
> at 
> org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
>
>         at 
> org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
>
>         at java.util.TimerThread.mainLoop(Timer.java:566)
>
>         at java.util.TimerThread.run(Timer.java:516)
>
> Below is my sites.xml file:
>
> <pool handle="cluster">
>
>     <execution provider="coaster" jobmanager="local:pbs" />
>
>     <profile namespace="globus" key="project">CI-SES000178</profile>
>
>     <profile namespace="globus" key="jobsPerNode">24</profile>
>
>     <profile namespace="globus" key="lowOverAllocation">100</profile>
>
>     <profile namespace="globus" key="highOverAllocation">100</profile>
>
>     <profile namespace="globus" 
> key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
>
>     <profile namespace="globus" key="maxtime">10800</profile>
>
>     <profile namespace="globus" key="maxWalltime">01:25:00</profile>
>
>     <profile namespace="globus" 
> key="userHomeOverride">/lustre/beagle2/mattshax/epsweep/swifthome</profile>
>
>     <profile namespace="globus" key="slots">20</profile>
>
>     <profile namespace="globus" key="maxnodes">600</profile>
>
>     <profile namespace="globus" key="nodeGranularity">1</profile>
>
>     <profile namespace="karajan" key="jobThrottle">180</profile>
>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>
>     <!-- <profile namespace="karajan" 
> key="workerLoggingLevel">trace</profile> -->
>
> <workdirectory>/dev/shm/mattshax/swiftapp</workdirectory>
>
>   </pool>
>
> MATTHEW SHAXTED
>
> SKIDMORE, OWINGS & MERRILL LLP
> 224 SOUTH MICHIGAN AVENUE
> CHICAGO, IL 60604
> T  (312) 360-4368
> MATTHEW.SHAXTED at SOM.COM <mailto:MATTHEW.SHAXTED at SOM.COM>
>
> cid:image001.png at 01CF9071.6FB46030 <http://www.som.com/>
>
> The information contained in this communication may be confidential, 
> is intended only for the use of the recipient(s) named above, and may 
> be legally privileged. If the reader of this message is not the 
> intended recipient, you are hereby notified that any dissemination, 
> distribution, or copying of this communication, or any of its 
> contents, is strictly prohibited and may be unlawful. If you have 
> received this communication in error, please return it to the 
> sen­­­der immediately and delete the original message and any copy of 
> it from your computer system. If you have any questions concerning 
> this message, please contact the sender.
>
> http://intranet.som.com/common/admin/file.cfm?f=%2Fresources%2Fcontent%2F5%2F0%2F4%2F4%2F6%2F4%2F0%2F3%2Fdocuments%2Fimagea560bf%2Egif%406e10073b%2E30854c37
>
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20150529/77b587cb/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 5311 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20150529/77b587cb/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 566 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20150529/77b587cb/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 5311 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20150529/77b587cb/attachment-0001.png>

More information about the Swift-user mailing list