Hi Mihael,<div><br></div><div>I tested this fix. It seems that the timeout issue for large-ish data and throttle > ~30 persists. I am not sure if this is data staging timeout though.</div><div><br></div><div>The setup that fails is as follows:</div>
<div><br></div><div>persistent coasters, resource= workers running on OSG</div>
<div>data size=8MB, 100 data items.</div><div>foreach throttle=40=jobthrottle.</div><div><br></div><div>The standard output seems intermittently showing some activity and then getting back to no activity without any progress on tasks.</div>
<div><br></div><div>Please find the log and stdouterr here: <a href="http://www.ci.uchicago.edu/~ketan/coaster-lab/std.out.err">http://www.ci.uchicago.edu/~ketan/coaster-lab/std.out.err</a>, <a href="http://www.ci.uchicago.edu/~ketan/coaster-lab/catsn-20110921-1535-v0t3gcg5.log">http://www.ci.uchicago.edu/~ketan/coaster-lab/catsn-20110921-1535-v0t3gcg5.log</a></div>
<div><br></div><div>When I tested with small data, 1MB, 2MB, 4MB, it did work. 4MB displayed a fat tail behavior though, ~94 tasks completing steadily and quickly while the last 5-6 tasks taking disproportionate times. The throttle in these cases was <= 30.</div>
<div><br></div><div><br></div><div>Regards,</div><div>Ketan</div><div><br><div class="gmail_quote">On Mon, Sep 12, 2011 at 7:19 PM, Mihael Hategan <span dir="ltr"><<a href="mailto:hategan@mcs.anl.gov" target="_blank">hategan@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Try now please (cog r3262).<br>
<div><br>
On Mon, 2011-09-12 at 15:56 -0500, Ketan Maheshwari wrote:<br>
</div><div><div></div><div>> Mihael,<br>
><br>
><br>
> I tried with the new <a href="http://worker.pl" target="_blank">worker.pl</a>, running a 100 task 10MB per task run<br>
> with throttle set at 100.<br>
><br>
><br>
> However, it seems to have failed with the same symptoms of timeout<br>
> error 521:<br>
><br>
><br>
> Caused by: null<br>
> Caused by:<br>
> org.globus.cog.abstraction.impl.common.execution.JobException: Job<br>
> failed with an exit code of 521<br>
> Progress: time: Mon, 12 Sep 2011 15:45:31 -0500 Submitted:53<br>
> Active:1 Failed:46<br>
> Progress: time: Mon, 12 Sep 2011 15:45:34 -0500 Submitted:53<br>
> Active:1 Failed:46<br>
> Exception in cat:<br>
> Arguments: [gpfs/pads/swift/ketan/indir10/data0002.txt]<br>
> Host: grid<br>
> Directory: catsn-20110912-1521-8jh2gar4/jobs/u/cat-u18visfk<br>
> - - -<br>
><br>
><br>
> Caused by: null<br>
> Caused by:<br>
> org.globus.cog.abstraction.impl.common.execution.JobException: Job<br>
> failed with an exit code of 521<br>
> Progress: time: Mon, 12 Sep 2011 15:45:45 -0500 Submitted:52<br>
> Active:1 Failed:47<br>
> Exception in cat:<br>
> Arguments: [gpfs/pads/swift/ketan/indir10/data0014.txt]<br>
> Host: grid<br>
> Directory: catsn-20110912-1521-8jh2gar4/jobs/x/cat-x18visfk<br>
><br>
><br>
> I had about 107 workers running at the time of these failures.<br>
><br>
><br>
> I started seeing the failure messages after about 20 minutes into this<br>
> run.<br>
><br>
><br>
> The logs are in <a href="http://www.ci.uchicago.edu/~ketan/pack.tgz" target="_blank">http://www.ci.uchicago.edu/~ketan/pack.tgz</a><br>
><br>
><br>
> Regards,<br>
> Ketan<br>
><br>
><br>
><br>
> On Mon, Sep 12, 2011 at 1:56 PM, Mihael Hategan <<a href="mailto:hategan@mcs.anl.gov" target="_blank">hategan@mcs.anl.gov</a>><br>
> wrote:<br>
> On Mon, 2011-09-12 at 11:58 -0500, Ketan Maheshwari wrote:<br>
><br>
> > After some discussion with Mike, Our conclusion from these<br>
> runs was<br>
> > that the parallel data transfers are causing timeouts from<br>
> the<br>
> > <a href="http://worker.pl" target="_blank">worker.pl</a>, further, we were undecided if somehow the timeout<br>
> threshold<br>
> > is set too agressive plus how are they determined and<br>
> whether a change<br>
> > in that value could resolve the issue.<br>
><br>
><br>
> Something like that. Worker.pl would use the time when a file<br>
> transfer<br>
> started to determine timeouts. This is undesirable. The<br>
> purpose of<br>
> timeouts is to determine whether the other side has stopped<br>
> from<br>
> properly following the flow of things. It follows that any<br>
> kind of<br>
> activity should reset the timeout... timer.<br>
><br>
> I updated the worker code to deal with the issue in a proper<br>
> way. But<br>
> now I need your help. This is perl code, and it needs testing.<br>
><br>
> So can you re-run, first with some simple test that uses<br>
> coaster staging<br>
> (just to make sure I didn't mess something up), and then the<br>
> version of<br>
> your tests that was most likely to fail?<br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> Ketan<br>
><br>
><br>
><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Ketan<br><br><br>
</div>