<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi Mike,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
I checked Jonathan's folders and it looks like the submit scripts and the PBS submit, submit.stdout and submit.stderr files</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">correctly were written under the runNNN/scripts folder. His latest run was using Swift-0.95-RC6 which failed with the logs</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">that you saw. The are also PBS*submit.stderr files which report the same "uninitialized value $s in concatenation" error.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">-Yadu</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 31, 2014 at 9:18 AM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@anl.gov" target="_blank">wilde@anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I see this from PBS in your home dir:<br>
<br>
blues$ cat <a href="http://583937.bmgt1.lcrc.anl.gov.ER" target="_blank">583937.bmgt1.lcrc.anl.gov.ER</a><br>
Use of uninitialized value $s in concatenation (.) or string at<br>
/home/ozik/.globus/coasters/<a href="http://cscript4312030037430783094.pl" target="_blank">cscript4312030037430783094.pl</a> line 2220.<br>
Use of uninitialized value $s in concatenation (.) or string at<br>
/home/ozik/.globus/coasters/<a href="http://cscript4312030037430783094.pl" target="_blank">cscript4312030037430783094.pl</a> line 2220.<br>
blues$<br>
<br>
That looks to me like a Swift bug in <a href="http://worker.pl" target="_blank">worker.pl</a><br>
<br>
We'll look into this angle.<br>
<br>
Also I'm curious why these files are not going into your run dir (but<br>
perhaps thats because youre running an older trunk release, not 0.95?<br>
Or, thats a separate 0.95 bug).<br>
<br>
- Mike<br>
<div class="HOEnZb"><div class="h5"><br>
On 7/31/14, 9:13 AM, Michael Wilde wrote:<br>
> Some discussion and diagnosis of this incident has taken place off list.<br>
><br>
> In a quick scan of the worker logs, I don't spot an obvious error that<br>
> would cause workers to exit.<br>
> Hopefully others on the Swift team can check those as well.<br>
><br>
> Jonathan, do you have stdout/err files from the PBS scheduler on blues,<br>
> in your runNNN log dirs?<br>
><br>
> If so, can you point us to them?<br>
><br>
> Thanks,<br>
><br>
> - Mike<br>
><br>
> On 7/29/14, 8:56 PM, Jonathan Ozik wrote:<br>
>> Hi all,<br>
>><br>
>> I’m getting spurious errors in the jobs that I’m running on Blues. The stdout includes exceptions like:<br>
>> exception @ swift-int.k, line: 511<br>
>> Caused by: Block task failed: Connection to worker lost<br>
>> java.io.IOException: Broken pipe<br>
>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)<br>
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)<br>
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)<br>
>> at sun.nio.ch.IOUtil.write(IOUtil.java:65)<br>
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)<br>
>> at org.globus.cog.coaster.channels.NIOSender.write(NIOSender.java:168)<br>
>> at org.globus.cog.coaster.channels.NIOSender.run(NIOSender.java:133)<br>
>><br>
>> These seem to occur at different parts of the submitted jobs. Let me know if there’s a log file that you’d like to look at.<br>
>><br>
>> In earlier attempts I was getting these warnings followed by broken pipe errors:<br>
>> Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000a0000000, 704643072, 2097152, 0) failed; error='Cannot allocate memory' (errno=12); Cannot allocate large pages, falling back to regular pages<br>
>><br>
>> Apparently that’s a known precursor of crashes on Java 7 as described here (<a href="http://www.oracle.com/technetwork/java/javase/7u51-relnotes-2085002.html" target="_blank">http://www.oracle.com/technetwork/java/javase/7u51-relnotes-2085002.html</a>):<br>
>> Area: hotspot/gc<br>
>> Synopsis: Crashes due to failure to allocate large pages.<br>
>><br>
>> On Linux, failures when allocating large pages can lead to crashes. When running JDK 7u51 or later versions, the issue can be recognized in two ways:<br>
>><br>
>> • Before the crash happens one or more lines similar to this will have been printed to the log:<br>
>> os::commit_memory(0x00000006b1600000, 352321536, 2097152, 0) failed;<br>
>> error='Cannot allocate memory' (errno=12); Cannot allocate large pages, falling back to regular pages<br>
>> • If a hs_err file is generated it will contain a line similar to this:<br>
>> Large page allocation failures have occurred 3 times<br>
>> The problem can be avoided by running with large page support turned off, for example by passing the "-XX:-UseLargePages" option to the java binary.<br>
>><br>
>> See 8007074 (not public).<br>
>><br>
>> So I added the -XX:-UseLargePages option in the invocations of Java code that I was responsible for. That seemed to get rid of the warning and the crashes for a while, but perhaps that was just a coincidence.<br>
>><br>
>> Jonathan<br>
>><br>
>> _______________________________________________<br>
>> Swift-user mailing list<br>
>> <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
>> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
<br>
--<br>
Michael Wilde<br>
Mathematics and Computer Science Computation Institute<br>
Argonne National Laboratory The University of Chicago<br>
<br>
_______________________________________________<br>
Swift-user mailing list<br>
<a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Yadu Nand B<div><br></div></div>
</div>