[Swift-devel] Coaster Task Submission Stalling

[Tim Armstrong]

I'm running a test Swift/T script that submit tasks to Coasters through the
C++ client and I'm seeing some odd behaviour where task
submission/execution is stalling for ~2 minute periods.  For example, I'm
seeing submit log messages like "submitting
urn:133-1409778135377-1409778135378: /bin/hostname" in bursts of several
seconds with a gap of roughly 2 minutes in between, e.g. I'm seeing bursts
with the following intervals in my logs.

16:07:04,603 to 16:07:10,391
16:09:07,377 to 16:09:13,076
16:11:10,005 to 16:11:16,770
16:13:13,291 to 16:13:19,296
16:15:16,000 to 16:15:21,602

>From what I can tell, the delay is on the coaster service side: the C
client is just waiting for a response.

The jobs are just being submitted through the local job manager, so I
wouldn't expect any delays there.  The tasks are also just "/bin/hostname",
so should return immediately.

I'm going to continue digging into this on my own, but the 2 minute delay
seems like a big clue: does anyone have an idea what could cause stalls in
task submission of 2 minute duration?

Cheers,
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140903/675e6857/attachment.html>