<div dir="ltr">For those interested in this problem, here is the latest heap plot of Jason's long (and still running) Beagle job.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 3, 2014 at 3:29 AM, David Kelly <span dir="ltr"><<a href="mailto:davidkelly@uchicago.edu" target="_blank">davidkelly@uchicago.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>I've spent the weekend working on the popdiagts script. I looked around on Geyser's filesystem and was able to find some input files that I can use. Once I found the data and got the 39 arguments correct, I was able to reproduce the problem. </div>
<div><br></div><div>I see a result that looks very similar to the initial report:</div><div><p>Progress: time: Mon, 03 Feb 2014 01:20:00 -0700 Active:1 Finished successfully:3</p><p>/glade/u/home/davkelly/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift: line 177: 31567 Killed java -Xmx8096M -XX:+HeapDumpOnOutOfMemoryError -Djava.endorsed.dirs=/glade/u/home/davkelly/swift-0.94/cog/modules/s...</p>
</div><div>To start, I ran Swift with the default of 1G heap size and within a few minutes I was able to see Swift being killed. A heap plot of a failing run:</div><div><br></div><div><a href="http://web.ci.uchicago.edu/~davidk/popdiagts-20140201-1458-i9hmaf0e.png" target="_blank">http://web.ci.uchicago.edu/~davidk/popdiagts-20140201-1458-i9hmaf0e.png</a><br>
</div><div><br></div><div>I tried bumping up the max heap size, but I ran into the same problem within a few minutes. The amount of memory used never seems to get very high. Here is a plot with 8G:</div><div><br></div><div>
<a href="http://web.ci.uchicago.edu/~davidk/popdiagts-20140203-0059-g6a11m24.png" target="_blank">http://web.ci.uchicago.edu/~davidk/popdiagts-20140203-0059-g6a11m24.png</a><br></div><div><br></div><div>I used jmap to generate several heap dumps during the run. They are about 100MB compressed, 400MB uncompressed, located at:<br>
</div><div><br></div><div><a href="http://web.ci.uchicago.edu/~davidk/heap1.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap1.gz</a><br></div><div><a href="http://web.ci.uchicago.edu/~davidk/heap2.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap2.gz</a><br>
</div><div><a href="http://web.ci.uchicago.edu/~davidk/heap3.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap3.gz</a><br></div><div><a href="http://web.ci.uchicago.edu/~davidk/heap4.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap4.gz</a><br>
</div><div><a href="http://web.ci.uchicago.edu/~davidk/heap5.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap5.gz</a><br></div><div><a href="http://web.ci.uchicago.edu/~davidk/heap6.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap6.gz</a><br>
</div><div><a href="http://web.ci.uchicago.edu/~davidk/heap7.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap7.gz</a><br></div><div><a href="http://web.ci.uchicago.edu/~davidk/heap8.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap8.gz</a><br>
</div><div><a href="http://web.ci.uchicago.edu/~davidk/heap9.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap9.gz</a><br></div><div><a href="http://web.ci.uchicago.edu/~davidk/heap10.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap10.gz</a><br>
</div><div><a href="http://web.ci.uchicago.edu/~davidk/heap11.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/heap11.gz</a><br></div><div><br></div><div>I used Eclipse Memory Analyzer to look at the heaps. You can view an html histogram of the objects at:</div>
<div><br></div><div><a href="http://web.ci.uchicago.edu/~davidk/heap-histogram/index.html" target="_blank">http://web.ci.uchicago.edu/~davidk/heap-histogram/index.html</a><br></div><div><br></div><div>It's possible that there was a sudden spike in memory at the end that the logs missed, but I don't think that's what's going on here.</div>
<div><br></div><div>As I was running the script, I opened top and saw the Swift CPU usage on the Geyser head node get extremely high, up to 700%. I think it's getting killed due to a kernel CPU throttle.</div><div><br>
</div><div>I went through the script line by line until I could narrow down where the problem was. I whittling away at it until I could get a small, readable, and data-independent test script that shows the problem.</div>
<div><br></div><div>Here it is:</div><div>----</div>type file;<br>app (file out) createFile() {<br> createFile @filename(out);<br>}<br><br>app (file out) createFileGivenArray (file fileArray[]) {<br> createFile @filename(out);<br>
}<br><br>file myArray[];<br>file myFile;<br><br>foreach f,i in [1:2] {<br> myArray[i] = createFile();<br>}<br><br>myFile = createFileGivenArray(myArray);<br>-----<div><br></div><div>On Midway you'll see the CPU usage on the head node jump to about 200% while the first app runs. If you repeat that pattern many times (like the original script does) you'll see CPU usage go even higher.</div>
<div><br></div><div>I've filed this as Bug 1195 ( <a href="https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1195" target="_blank">https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1195</a> ). The package to reproduce this is at <a href="http://web.ci.uchicago.edu/~davidk/popdiag.tar.gz" target="_blank">http://web.ci.uchicago.edu/~davidk/popdiag.tar.gz</a>.</div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div class="h5">On Tue, Jan 28, 2014 at 3:29 PM, Wilde, Michael J. <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
<div>
<div style="direction:ltr;font-size:10pt;font-family:Tahoma">
<blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div style="font-size:16px;font-family:Times New Roman">
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>From:</b></font></div>
David Kelly [<a href="mailto:davidkelly@uchicago.edu" target="_blank">davidkelly@uchicago.edu</a>]</div>
<div style="font-size:16px;font-family:Times New Roman">
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>Sent:</b></font></div>
Tuesday, January 28, 2014 2:47 PM</div>
<div>
<div style="direction:ltr"><font><b>...</b></font></div>
</div><div>
<div style="font-size:16px;font-family:Times New Roman">
<div>
<div dir="ltr">I don't have too many updates on Sheri's problem. I was able to run the older standalone example I had on Geyser and did not see any issues with excessive amounts of resident memory being used. </div>
</div>
</div>
</div><div style="font-size:16px;font-family:Times New Roman">
<div>
<div dir="ltr">...</div>
</div>
</div>
</blockquote>
<div style="font-size:16px;font-family:Times New Roman">
<div></div>
<div>
<div dir="ltr"><br>
</div>
<div dir="ltr">I think the failure was exceeding the Java heap size, not an RSS problem, right?<br>
<div><br>
</div>
</div>
</div>
</div><div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div style="font-size:16px;font-family:Times New Roman">
<div>
<div dir="ltr">
<div>I think we might be better off shifting the way we approach this problem. It's difficult to run these apps, and to run them in the same way the users do. There's also a long delay getting responses. I think we'd be better off focusing on adding comprehensive
memory tests to the test suite, measuring, plotting, and then documenting solutions/strategies into the user guide. It will take some time, but I think it's the best approach since everything would be under our own control, and it would provide solutions for
all users.</div>
</div>
</div>
</div>
<div style="font-size:16px;font-family:Times New Roman">
<div>
<div dir="ltr">
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
</div><div style="font-size:16px;font-family:Times New Roman">
<div>
<div dir="ltr">
<div>That sounds good, while we are waiting for debugging info from users. But we should still strive to reproduce problems that users are encountering, and on giving them code updates with additional debugging hooks or possible remedies to test.</div>
<div><br>
</div>
</div>
<div class="gmail_extra">- Mike<div><br>
<br>
<div class="gmail_quote">On Tue, Jan 28, 2014 at 12:36 PM, Wilde, Michael J. <span dir="ltr">
<<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-size:10pt;font-family:Tahoma">Yadu, David, can you send updates on this to Swift devel, and lets talk this afternoon at 3PM to discuss?
<div><br>
</div>
<div>Thanks,<br>
<div><br>
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div>- Mike </div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div></div>
</div>
</div>
</div>
</div>
<br></div></div>_______________________________________________<br>
Swift-devel mailing list<br>
<a href="mailto:Swift-devel@ci.uchicago.edu" target="_blank">Swift-devel@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>