<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi,<br>
Nika can probably be more specific, but the last time we ran the 244
molecule MolDyn, the workflow failed on the last few jobs, and the
failures were application specific, not Swift or Falkon. I believe the
specific issue that caused those jobs to fail has been resolved. <br>
<br>
We have made another attempt at the MolDyn 244 molecule run, and from
what I can tell, it did not complete successfully again. We were
supposed to have 20497 jobs...<br>
<br>
<table x:str="" style="border-collapse: collapse; width: 144pt;"
border="0" cellpadding="0" cellspacing="0" width="192">
<col style="width: 48pt;" span="3" width="64"> <tbody>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt; width: 48pt;" x:num="" align="right"
height="17" width="64">1</td>
<td style="width: 48pt;" x:num="" align="right" width="64">1</td>
<td style="width: 48pt;" x:num="" x:fmla="=A1*B1" align="right"
width="64">1</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" x:num="" align="right" height="17">1</td>
<td x:num="" align="right">244</td>
<td x:num="" x:fmla="=A2*B2" align="right">244</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" x:num="" align="right" height="17">1</td>
<td x:num="" align="right">244</td>
<td x:num="" x:fmla="=A3*B3" align="right">244</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" x:num="" align="right" height="17">68</td>
<td x:num="" align="right">244</td>
<td x:num="" x:fmla="=A4*B4" align="right">16592</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" x:num="" align="right" height="17">1</td>
<td x:num="" align="right">244</td>
<td x:num="" x:fmla="=A5*B5" align="right">244</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" x:num="" align="right" height="17">11</td>
<td x:num="" align="right">244</td>
<td x:num="" x:fmla="=A6*B6" align="right">2684</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" x:num="" align="right" height="17">1</td>
<td x:num="" align="right">244</td>
<td x:num="" x:fmla="=A7*B7" align="right">244</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" x:num="" align="right" height="17">1</td>
<td x:num="" align="right">244</td>
<td x:num="" x:fmla="=A8*B8" align="right">244</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" height="17"><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr style="height: 12.75pt;" height="17">
<td style="height: 12.75pt;" height="17"><br>
</td>
<td><br>
</td>
<td x:num="" x:fmla="=SUM(C1:C9)" align="right">20497</td>
</tr>
</tbody>
</table>
<br>
but we have:<br>
20482 with exit code 0<br>
1 with exit code -3<br>
2 with exit code 253<br>
<br>
I forgot to enable the debug at the workers, so I don't know what the
STDOUT and STDERR was for these 3 jobs. Given that Swift retries 3
times a job before it fails the workflow, my guess is that these 3 jobs
were really the same job failing 3 times. The failure occurred on 3
different machines, so I don't think it was machine related. Nika, can
you tell from the various Swift logs what happened to these 3 jobs? Is
this the same issue as we had on the last 244 mol run? It looks like
we failed the workflow with 15 jobs to go. <br>
<br>
The graphs all look nice, similar to the last ones we had. If people
really want to see them, I can generate them again. Otherwise, look at
<a class="moz-txt-link-freetext" href="http://tg-viz-login1.uc.teragrid.org:51000/index.htm">http://tg-viz-login1.uc.teragrid.org:51000/index.htm</a> to see the last
10K samples of the experiment.<br>
<br>
Nika, after you try to figure out what happened, can you simply retry
the workflow, maybe it will manage to finish the last 15 jobs.
Depending on what problem we find, I think we might conclude that 3
retries is not enough, and we might want to have a higher number as the
default when running with Falkon. If the error was an application
error, then no matter how many retries we have, it won't make any
difference.<br>
<br>
Ioan<br>
<br>
<br>
<br>
Michael Wilde wrote:
<blockquote cite="mid:46B3B367.6090701@mcs.anl.gov" type="cite">Im
catching up from some of this weeks email.
<br>
<br>
I didnt see a followup to this, nor can I tell which two jobs Ian is
referring to or where those came from. Can anyone clarify what this
issue is here?
<br>
<br>
<br>
Ian Foster wrote:
<br>
<blockquote type="cite">Hi,
<br>
<br>
I am curious whether we found out why those two jobs (?) were failing
at the end of the big MolDyn run?
<br>
<br>
Ian.
<br>
<br>
_______________________________________________
<br>
Swift-devel mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a>
<br>
<br>
<br>
</blockquote>
_______________________________________________
<br>
Swift-devel mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a>
<br>
<br>
</blockquote>
</body>
</html>