From wilde at mcs.anl.gov Sun Feb 2 19:39:23 2014 From: wilde at mcs.anl.gov (Wilde, Michael J.) Date: Mon, 3 Feb 2014 01:39:23 +0000 Subject: [Swift-devel] Adding versions in bugzilla In-Reply-To: References: Message-ID: <85C85E44DD880E498CEA5A501B27954BEA47FE@DITKA.anl.gov> Justin or Mihael, can you advise on this? We should at same time clean up our bugzilla defaults. Less junk fields, ideally. Thanks, - Mike -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago ________________________________ From: swift-devel-bounces at ci.uchicago.edu [swift-devel-bounces at ci.uchicago.edu] on behalf of David Kelly [davidkelly at uchicago.edu] Sent: Monday, January 20, 2014 11:06 AM To: swift-devel Subject: [Swift-devel] Adding versions in bugzilla Hello, Does anyone know how to add swift releases to the list of "versions" in bugzilla? I'd like to add a 0.95 version there, but I don't see any options to do that (and not sure if I have the permissions to do that or not). Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidkelly at uchicago.edu Mon Feb 3 03:29:57 2014 From: davidkelly at uchicago.edu (David Kelly) Date: Mon, 3 Feb 2014 03:29:57 -0600 Subject: [Swift-devel] Progress on Swift RAM usage problem? In-Reply-To: <85C85E44DD880E498CEA5A501B27954BEA2486@DITKA.anl.gov> References: <85C85E44DD880E498CEA5A501B27954BEA237C@DITKA.anl.gov> <85C85E44DD880E498CEA5A501B27954BEA2486@DITKA.anl.gov> Message-ID: Hello, I've spent the weekend working on the popdiagts script. I looked around on Geyser's filesystem and was able to find some input files that I can use. Once I found the data and got the 39 arguments correct, I was able to reproduce the problem. I see a result that looks very similar to the initial report: Progress: time: Mon, 03 Feb 2014 01:20:00 -0700 Active:1 Finished successfully:3 /glade/u/home/davkelly/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift: line 177: 31567 Killed java -Xmx8096M -XX:+HeapDumpOnOutOfMemoryError -Djava.endorsed.dirs=/glade/u/home/davkelly/swift-0.94/cog/modules/s... To start, I ran Swift with the default of 1G heap size and within a few minutes I was able to see Swift being killed. A heap plot of a failing run: http://web.ci.uchicago.edu/~davidk/popdiagts-20140201-1458-i9hmaf0e.png I tried bumping up the max heap size, but I ran into the same problem within a few minutes. The amount of memory used never seems to get very high. Here is a plot with 8G: http://web.ci.uchicago.edu/~davidk/popdiagts-20140203-0059-g6a11m24.png I used jmap to generate several heap dumps during the run. They are about 100MB compressed, 400MB uncompressed, located at: http://web.ci.uchicago.edu/~davidk/heap1.gz http://web.ci.uchicago.edu/~davidk/heap2.gz http://web.ci.uchicago.edu/~davidk/heap3.gz http://web.ci.uchicago.edu/~davidk/heap4.gz http://web.ci.uchicago.edu/~davidk/heap5.gz http://web.ci.uchicago.edu/~davidk/heap6.gz http://web.ci.uchicago.edu/~davidk/heap7.gz http://web.ci.uchicago.edu/~davidk/heap8.gz http://web.ci.uchicago.edu/~davidk/heap9.gz http://web.ci.uchicago.edu/~davidk/heap10.gz http://web.ci.uchicago.edu/~davidk/heap11.gz I used Eclipse Memory Analyzer to look at the heaps. You can view an html histogram of the objects at: http://web.ci.uchicago.edu/~davidk/heap-histogram/index.html It's possible that there was a sudden spike in memory at the end that the logs missed, but I don't think that's what's going on here. As I was running the script, I opened top and saw the Swift CPU usage on the Geyser head node get extremely high, up to 700%. I think it's getting killed due to a kernel CPU throttle. I went through the script line by line until I could narrow down where the problem was. I whittling away at it until I could get a small, readable, and data-independent test script that shows the problem. Here it is: ---- type file; app (file out) createFile() { createFile @filename(out); } app (file out) createFileGivenArray (file fileArray[]) { createFile @filename(out); } file myArray[]; file myFile; foreach f,i in [1:2] { myArray[i] = createFile(); } myFile = createFileGivenArray(myArray); ----- On Midway you'll see the CPU usage on the head node jump to about 200% while the first app runs. If you repeat that pattern many times (like the original script does) you'll see CPU usage go even higher. I've filed this as Bug 1195 ( https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1195 ). The package to reproduce this is at http://web.ci.uchicago.edu/~davidk/popdiag.tar.gz. On Tue, Jan 28, 2014 at 3:29 PM, Wilde, Michael J. wrote: > *From:* > David Kelly [davidkelly at uchicago.edu] > *Sent:* > Tuesday, January 28, 2014 2:47 PM > *...* > I don't have too many updates on Sheri's problem. I was able to run the > older standalone example I had on Geyser and did not see any issues with > excessive amounts of resident memory being used. > ... > > > I think the failure was exceeding the Java heap size, not an RSS > problem, right? > > I think we might be better off shifting the way we approach this > problem. It's difficult to run these apps, and to run them in the same way > the users do. There's also a long delay getting responses. I think we'd be > better off focusing on adding comprehensive memory tests to the test suite, > measuring, plotting, and then documenting solutions/strategies into the > user guide. It will take some time, but I think it's the best approach > since everything would be under our own control, and it would provide > solutions for all users. > > That sounds good, while we are waiting for debugging info from users. > But we should still strive to reproduce problems that users are > encountering, and on giving them code updates with additional debugging > hooks or possible remedies to test. > > - Mike > > > On Tue, Jan 28, 2014 at 12:36 PM, Wilde, Michael J. wrote: > >> Yadu, David, can you send updates on this to Swift devel, and lets talk >> this afternoon at 3PM to discuss? >> >> Thanks, >> >> - Mike >> >> > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidkelly at uchicago.edu Tue Feb 4 11:44:35 2014 From: davidkelly at uchicago.edu (David Kelly) Date: Tue, 4 Feb 2014 11:44:35 -0600 Subject: [Swift-devel] Migrating to github Message-ID: Right now we have two repos - cog on sourceforge, and Swift on svn.ci.uchicago.edu. Sourceforge's SVN commit hooks are broken, so we do not get email updates automatically from them when a change is made. Instead I rigged a cron job that runs every 5 minutes, checks for changes, and emails them out. It's a little ugly. Developers wanting to make contributions to Swift need a CI account. This may discourage new developers from getting involved. Having two repos makes branching, tagging, and building more tedious than it needs to be. This may need more discussion, but I would like to propose that starting with 0.95, let's merge cog and Swift and move to github. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Tue Feb 4 12:00:54 2014 From: wilde at mcs.anl.gov (Wilde, Michael J.) Date: Tue, 4 Feb 2014 18:00:54 +0000 Subject: [Swift-devel] Migrating to github In-Reply-To: References: Message-ID: <85C85E44DD880E498CEA5A501B27954BEA4D27@DITKA.anl.gov> In in favor of considering this, and putting Swifts /K and /T in close proximity in the repo. I think its a step to becoming a more open open-source project. - Mike -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago ________________________________ From: swift-devel-bounces at ci.uchicago.edu [swift-devel-bounces at ci.uchicago.edu] on behalf of David Kelly [davidkelly at uchicago.edu] Sent: Tuesday, February 04, 2014 11:44 AM To: swift-devel Subject: [Swift-devel] Migrating to github Right now we have two repos - cog on sourceforge, and Swift on svn.ci.uchicago.edu. Sourceforge's SVN commit hooks are broken, so we do not get email updates automatically from them when a change is made. Instead I rigged a cron job that runs every 5 minutes, checks for changes, and emails them out. It's a little ugly. Developers wanting to make contributions to Swift need a CI account. This may discourage new developers from getting involved. Having two repos makes branching, tagging, and building more tedious than it needs to be. This may need more discussion, but I would like to propose that starting with 0.95, let's merge cog and Swift and move to github. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yadudoc1729 at gmail.com Tue Feb 4 13:16:53 2014 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Tue, 4 Feb 2014 13:16:53 -0600 Subject: [Swift-devel] Migrating to github In-Reply-To: <85C85E44DD880E498CEA5A501B27954BEA4D27@DITKA.anl.gov> References: <85C85E44DD880E498CEA5A501B27954BEA4D27@DITKA.anl.gov> Message-ID: I think moving to Github is a great idea, also makes sense to merge cog and swift into one repo. Part of the reason for the popularity of Github is the social/open aspect of development. Going forward we'd definitely need that, and the sooner we migrate, the better. It also looks like there's some nice documentation here http://git-scm.com/book/ch8-1.html about git-svn interoperability. -Yadu On Tue, Feb 4, 2014 at 12:00 PM, Wilde, Michael J. wrote: > In in favor of considering this, and putting Swifts /K and /T in close > proximity in the repo. > > I think its a step to becoming a more open open-source project. > > - Mike > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > ________________________________ > From: swift-devel-bounces at ci.uchicago.edu > [swift-devel-bounces at ci.uchicago.edu] on behalf of David Kelly > [davidkelly at uchicago.edu] > Sent: Tuesday, February 04, 2014 11:44 AM > To: swift-devel > Subject: [Swift-devel] Migrating to github > > Right now we have two repos - cog on sourceforge, and Swift on > svn.ci.uchicago.edu. > > Sourceforge's SVN commit hooks are broken, so we do not get email updates > automatically from them when a change is made. Instead I rigged a cron job > that runs every 5 minutes, checks for changes, and emails them out. It's a > little ugly. > > Developers wanting to make contributions to Swift need a CI account. This > may discourage new developers from getting involved. > > Having two repos makes branching, tagging, and building more tedious than it > needs to be. > > This may need more discussion, but I would like to propose that starting > with 0.95, let's merge cog and Swift and move to github. > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -- Yadu Nand B From ketan at mcs.anl.gov Tue Feb 4 13:32:24 2014 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Tue, 4 Feb 2014 13:32:24 -0600 Subject: [Swift-devel] Migrating to github In-Reply-To: References: <85C85E44DD880E498CEA5A501B27954BEA4D27@DITKA.anl.gov> Message-ID: Great idea. However, one thing to consider is that there will still be need of svn repos such as SwiftApps and ExM which are too big to migrate to github. So there will be two repo/protocols to manage. On Tue, Feb 4, 2014 at 1:16 PM, Yadu Nand wrote: > I think moving to Github is a great idea, also makes sense to merge > cog and swift into one repo. > > Part of the reason for the popularity of Github is the social/open > aspect of development. Going > forward we'd definitely need that, and the sooner we migrate, the better. > > It also looks like there's some nice documentation here > http://git-scm.com/book/ch8-1.html > about git-svn interoperability. > > -Yadu > > On Tue, Feb 4, 2014 at 12:00 PM, Wilde, Michael J. > wrote: > > In in favor of considering this, and putting Swifts /K and /T in close > > proximity in the repo. > > > > I think its a step to becoming a more open open-source project. > > > > - Mike > > -- > > Michael Wilde > > Mathematics and Computer Science Computation Institute > > Argonne National Laboratory The University of Chicago > > > > ________________________________ > > From: swift-devel-bounces at ci.uchicago.edu > > [swift-devel-bounces at ci.uchicago.edu] on behalf of David Kelly > > [davidkelly at uchicago.edu] > > Sent: Tuesday, February 04, 2014 11:44 AM > > To: swift-devel > > Subject: [Swift-devel] Migrating to github > > > > Right now we have two repos - cog on sourceforge, and Swift on > > svn.ci.uchicago.edu. > > > > Sourceforge's SVN commit hooks are broken, so we do not get email updates > > automatically from them when a change is made. Instead I rigged a cron > job > > that runs every 5 minutes, checks for changes, and emails them out. It's > a > > little ugly. > > > > Developers wanting to make contributions to Swift need a CI account. This > > may discourage new developers from getting involved. > > > > Having two repos makes branching, tagging, and building more tedious > than it > > needs to be. > > > > This may need more discussion, but I would like to propose that starting > > with 0.95, let's merge cog and Swift and move to github. > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > -- > Yadu Nand B > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Feb 4 13:44:16 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 04 Feb 2014 11:44:16 -0800 Subject: [Swift-devel] Adding versions in bugzilla In-Reply-To: <85C85E44DD880E498CEA5A501B27954BEA47FE@DITKA.anl.gov> References: <85C85E44DD880E498CEA5A501B27954BEA47FE@DITKA.anl.gov> Message-ID: <1391543056.6232.4.camel@echo> Sorry for the delay. Bugzilla has a link at the top called "Administration". On the target page, there is a link called "Products" where you can eventually edit version numbers. It requires admin rights. Current admins are Ben, I, Skenny, Mike, and Justin. I made David an admin. He can now add swift versions and pretty much everything. Mihael On Mon, 2014-02-03 at 01:39 +0000, Wilde, Michael J. wrote: > Justin or Mihael, can you advise on this? > > We should at same time clean up our bugzilla defaults. Less junk fields, ideally. > > Thanks, > > - Mike > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > ________________________________ > From: swift-devel-bounces at ci.uchicago.edu [swift-devel-bounces at ci.uchicago.edu] on behalf of David Kelly [davidkelly at uchicago.edu] > Sent: Monday, January 20, 2014 11:06 AM > To: swift-devel > Subject: [Swift-devel] Adding versions in bugzilla > > Hello, > > Does anyone know how to add swift releases to the list of "versions" in bugzilla? I'd like to add a 0.95 version there, but I don't see any options to do that (and not sure if I have the permissions to do that or not). > > Thanks, > David > Justin or Mihael, can you advise on this? > > > We should at same time clean up our bugzilla defaults. Less junk > fields, ideally. > > > Thanks, > > - Mike > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of > Chicago > > > > ______________________________________________________________________ > From: swift-devel-bounces at ci.uchicago.edu > [swift-devel-bounces at ci.uchicago.edu] on behalf of David Kelly > [davidkelly at uchicago.edu] > Sent: Monday, January 20, 2014 11:06 AM > To: swift-devel > Subject: [Swift-devel] Adding versions in bugzilla > > > > Hello, > > > Does anyone know how to add swift releases to the list of "versions" > in bugzilla? I'd like to add a 0.95 version there, but I don't see any > options to do that (and not sure if I have the permissions to do that > or not). > > Thanks, > David > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From wozniak at mcs.anl.gov Tue Feb 4 13:44:37 2014 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Tue, 4 Feb 2014 13:44:37 -0600 Subject: [Swift-devel] Migrating to github In-Reply-To: References: <85C85E44DD880E498CEA5A501B27954BEA4D27@DITKA.anl.gov> Message-ID: <52F14325.8010805@mcs.anl.gov> We can just move the Swift/T code, not the whole repo. On 02/04/2014 01:32 PM, Ketan Maheshwari wrote: > Great idea. However, one thing to consider is that there will still be > need of svn repos such as SwiftApps and ExM which are too big to > migrate to github. So there will be two repo/protocols to manage. > > > On Tue, Feb 4, 2014 at 1:16 PM, Yadu Nand > wrote: > > I think moving to Github is a great idea, also makes sense to merge > cog and swift into one repo. > > Part of the reason for the popularity of Github is the social/open > aspect of development. Going > forward we'd definitely need that, and the sooner we migrate, the > better. > > It also looks like there's some nice documentation here > http://git-scm.com/book/ch8-1.html > about git-svn interoperability. > > -Yadu > > On Tue, Feb 4, 2014 at 12:00 PM, Wilde, Michael J. > > wrote: > > In in favor of considering this, and putting Swifts /K and /T in > close > > proximity in the repo. > > > > I think its a step to becoming a more open open-source project. > > > > - Mike > > -- > > Michael Wilde > > Mathematics and Computer Science Computation Institute > > Argonne National Laboratory The University of > Chicago > > > > ________________________________ > > From: swift-devel-bounces at ci.uchicago.edu > > > [swift-devel-bounces at ci.uchicago.edu > ] on behalf of David Kelly > > [davidkelly at uchicago.edu ] > > Sent: Tuesday, February 04, 2014 11:44 AM > > To: swift-devel > > Subject: [Swift-devel] Migrating to github > > > > Right now we have two repos - cog on sourceforge, and Swift on > > svn.ci.uchicago.edu . > > > > Sourceforge's SVN commit hooks are broken, so we do not get > email updates > > automatically from them when a change is made. Instead I rigged > a cron job > > that runs every 5 minutes, checks for changes, and emails them > out. It's a > > little ugly. > > > > Developers wanting to make contributions to Swift need a CI > account. This > > may discourage new developers from getting involved. > > > > Having two repos makes branching, tagging, and building more > tedious than it > > needs to be. > > > > This may need more discussion, but I would like to propose that > starting > > with 0.95, let's merge cog and Swift and move to github. > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > -- > Yadu Nand B > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Justin M Wozniak -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Feb 4 14:04:47 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 04 Feb 2014 12:04:47 -0800 Subject: [Swift-devel] Migrating to github In-Reply-To: References: Message-ID: <1391544287.6232.11.camel@echo> Interesting. Given that we are the only active committers to java cog at this point, I see no downside to making our own copy and merging it with swift. I also believe that the license allows this, but we need to check the details. Might seem a little backstabby towards Gregor, but I'm not sure how much he cares at this point. I can send him an email and explain that it would make it easier for us to do things this way. I suspect that a prominent "this project uses code developed by ..." would be a reasonable way to address the moral/ethical issues involved. Mihael On Tue, 2014-02-04 at 11:44 -0600, David Kelly wrote: > Right now we have two repos - cog on sourceforge, and Swift on > svn.ci.uchicago.edu. > > Sourceforge's SVN commit hooks are broken, so we do not get email updates > automatically from them when a change is made. Instead I rigged a cron job > that runs every 5 minutes, checks for changes, and emails them out. It's a > little ugly. > > Developers wanting to make contributions to Swift need a CI account. This > may discourage new developers from getting involved. > > Having two repos makes branching, tagging, and building more tedious than > it needs to be. > > This may need more discussion, but I would like to propose that starting > with 0.95, let's merge cog and Swift and move to github. > Right now we have two repos - cog on sourceforge, and Swift on > svn.ci.uchicago.edu. > > > Sourceforge's SVN commit hooks are broken, so we do not get email > updates automatically from them when a change is made. Instead I > rigged a cron job that runs every 5 minutes, checks for changes, and > emails them out. It's a little ugly. > > > Developers wanting to make contributions to Swift need a CI account. > This may discourage new developers from getting involved. > > > Having two repos makes branching, tagging, and building more tedious > than it needs to be. > > > This may need more discussion, but I would like to propose that > starting with 0.95, let's merge cog and Swift and move to github. > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From tim.g.armstrong at gmail.com Tue Feb 4 16:59:04 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Tue, 4 Feb 2014 16:59:04 -0600 Subject: [Swift-devel] Migrating to github In-Reply-To: <1391544287.6232.11.camel@echo> References: <1391544287.6232.11.camel@echo> Message-ID: I actually have a version of Swift/T up on github: https://github.com/timarmstrong/swift-t It was a minor pain to get it set up, but I have scripts now that I can run to copy across changes from the SVN repo. - Tim On Tue, Feb 4, 2014 at 2:04 PM, Mihael Hategan wrote: > Interesting. > > Given that we are the only active committers to java cog at this point, > I see no downside to making our own copy and merging it with swift. > > I also believe that the license allows this, but we need to check the > details. > > Might seem a little backstabby towards Gregor, but I'm not sure how much > he cares at this point. I can send him an email and explain that it > would make it easier for us to do things this way. I suspect that a > prominent "this project uses code developed by ..." would be a > reasonable way to address the moral/ethical issues involved. > > Mihael > > On Tue, 2014-02-04 at 11:44 -0600, David Kelly wrote: > > Right now we have two repos - cog on sourceforge, and Swift on > > svn.ci.uchicago.edu. > > > > Sourceforge's SVN commit hooks are broken, so we do not get email updates > > automatically from them when a change is made. Instead I rigged a cron > job > > that runs every 5 minutes, checks for changes, and emails them out. It's > a > > little ugly. > > > > Developers wanting to make contributions to Swift need a CI account. This > > may discourage new developers from getting involved. > > > > Having two repos makes branching, tagging, and building more tedious than > > it needs to be. > > > > This may need more discussion, but I would like to propose that starting > > with 0.95, let's merge cog and Swift and move to github. > > Right now we have two repos - cog on sourceforge, and Swift on > > svn.ci.uchicago.edu. > > > > > > Sourceforge's SVN commit hooks are broken, so we do not get email > > updates automatically from them when a change is made. Instead I > > rigged a cron job that runs every 5 minutes, checks for changes, and > > emails them out. It's a little ugly. > > > > > > Developers wanting to make contributions to Swift need a CI account. > > This may discourage new developers from getting involved. > > > > > > Having two repos makes branching, tagging, and building more tedious > > than it needs to be. > > > > > > This may need more discussion, but I would like to propose that > > starting with 0.95, let's merge cog and Swift and move to github. > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidkelly at uchicago.edu Fri Feb 7 10:38:46 2014 From: davidkelly at uchicago.edu (David Kelly) Date: Fri, 7 Feb 2014 10:38:46 -0600 Subject: [Swift-devel] Progress on Swift RAM usage problem? In-Reply-To: References: <85C85E44DD880E498CEA5A501B27954BEA237C@DITKA.anl.gov> <85C85E44DD880E498CEA5A501B27954BEA2486@DITKA.anl.gov> Message-ID: For those interested in this problem, here is the latest heap plot of Jason's long (and still running) Beagle job. On Mon, Feb 3, 2014 at 3:29 AM, David Kelly wrote: > Hello, > > I've spent the weekend working on the popdiagts script. I looked around on > Geyser's filesystem and was able to find some input files that I can use. > Once I found the data and got the 39 arguments correct, I was able to > reproduce the problem. > > I see a result that looks very similar to the initial report: > > Progress: time: Mon, 03 Feb 2014 01:20:00 -0700 Active:1 Finished > successfully:3 > > /glade/u/home/davkelly/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift: > line 177: 31567 Killed java -Xmx8096M > -XX:+HeapDumpOnOutOfMemoryError > -Djava.endorsed.dirs=/glade/u/home/davkelly/swift-0.94/cog/modules/s... > To start, I ran Swift with the default of 1G heap size and within a few > minutes I was able to see Swift being killed. A heap plot of a failing run: > > http://web.ci.uchicago.edu/~davidk/popdiagts-20140201-1458-i9hmaf0e.png > > I tried bumping up the max heap size, but I ran into the same problem > within a few minutes. The amount of memory used never seems to get very > high. Here is a plot with 8G: > > http://web.ci.uchicago.edu/~davidk/popdiagts-20140203-0059-g6a11m24.png > > I used jmap to generate several heap dumps during the run. They are about > 100MB compressed, 400MB uncompressed, located at: > > http://web.ci.uchicago.edu/~davidk/heap1.gz > http://web.ci.uchicago.edu/~davidk/heap2.gz > http://web.ci.uchicago.edu/~davidk/heap3.gz > http://web.ci.uchicago.edu/~davidk/heap4.gz > http://web.ci.uchicago.edu/~davidk/heap5.gz > http://web.ci.uchicago.edu/~davidk/heap6.gz > http://web.ci.uchicago.edu/~davidk/heap7.gz > http://web.ci.uchicago.edu/~davidk/heap8.gz > http://web.ci.uchicago.edu/~davidk/heap9.gz > http://web.ci.uchicago.edu/~davidk/heap10.gz > http://web.ci.uchicago.edu/~davidk/heap11.gz > > I used Eclipse Memory Analyzer to look at the heaps. You can view an html > histogram of the objects at: > > http://web.ci.uchicago.edu/~davidk/heap-histogram/index.html > > It's possible that there was a sudden spike in memory at the end that the > logs missed, but I don't think that's what's going on here. > > As I was running the script, I opened top and saw the Swift CPU usage on > the Geyser head node get extremely high, up to 700%. I think it's getting > killed due to a kernel CPU throttle. > > I went through the script line by line until I could narrow down where the > problem was. I whittling away at it until I could get a small, readable, > and data-independent test script that shows the problem. > > Here it is: > ---- > type file; > app (file out) createFile() { > createFile @filename(out); > } > > app (file out) createFileGivenArray (file fileArray[]) { > createFile @filename(out); > } > > file myArray[]; > file myFile; > > foreach f,i in [1:2] { > myArray[i] = createFile(); > } > > myFile = createFileGivenArray(myArray); > ----- > > On Midway you'll see the CPU usage on the head node jump to about 200% > while the first app runs. If you repeat that pattern many times (like the > original script does) you'll see CPU usage go even higher. > > I've filed this as Bug 1195 ( > https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1195 ). The package to > reproduce this is at http://web.ci.uchicago.edu/~davidk/popdiag.tar.gz. > > > > On Tue, Jan 28, 2014 at 3:29 PM, Wilde, Michael J. wrote: > >> *From:* >> David Kelly [davidkelly at uchicago.edu] >> *Sent:* >> Tuesday, January 28, 2014 2:47 PM >> *...* >> I don't have too many updates on Sheri's problem. I was able to run >> the older standalone example I had on Geyser and did not see any issues >> with excessive amounts of resident memory being used. >> ... >> >> >> I think the failure was exceeding the Java heap size, not an RSS >> problem, right? >> >> I think we might be better off shifting the way we approach this >> problem. It's difficult to run these apps, and to run them in the same way >> the users do. There's also a long delay getting responses. I think we'd be >> better off focusing on adding comprehensive memory tests to the test suite, >> measuring, plotting, and then documenting solutions/strategies into the >> user guide. It will take some time, but I think it's the best approach >> since everything would be under our own control, and it would provide >> solutions for all users. >> >> That sounds good, while we are waiting for debugging info from >> users. But we should still strive to reproduce problems that users are >> encountering, and on giving them code updates with additional debugging >> hooks or possible remedies to test. >> >> - Mike >> >> >> On Tue, Jan 28, 2014 at 12:36 PM, Wilde, Michael J. wrote: >> >>> Yadu, David, can you send updates on this to Swift devel, and lets >>> talk this afternoon at 3PM to discuss? >>> >>> Thanks, >>> >>> - Mike >>> >>> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: heap-plot.png Type: image/png Size: 12661 bytes Desc: not available URL: From ketan at mcs.anl.gov Mon Feb 10 12:56:20 2014 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Mon, 10 Feb 2014 12:56:20 -0600 Subject: [Swift-devel] single quotes in arg Message-ID: Hi, A user reported that Swift was not recognizing single quotes within the text of a commandline arg. I was able to reproduce the issue: Swift source (printargs.swift) : string anarg=@arg("filter_cmd"); trace(anarg); $ swift printargs.swift -filter_cmd="'hello' world" Progress: time: Mon, 10 Feb 2014 12:47:41 -0600 SwiftScript trace: hello world Escape chars do not work: $ swift printargs.swift -filter_cmd=" \'hello\' world" /Users/ketan/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift: eval: line 177: unexpected EOF while looking for matching `'' /Users/ketan/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift: eval: line 178: syntax error: unexpected end of file Single quotes in the outer string do not work: $ swift printargs.swift -filter_cmd=' 'hello' world' Progress: time: Mon, 10 Feb 2014 12:49:59 -0600 SwiftScript trace: hello world Outer single quotes and inner double quotes seem to be working: $ swift printargs.swift -filter_cmd=' "hello" world' Progress: time: Mon, 10 Feb 2014 12:51:40 -0600 SwiftScript trace: "hello" world A single single quote does not work: swift printargs.swift -filter_cmd=" he'llo world" /Users/ketan/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift: eval: line 177: unexpected EOF while looking for matching `'' /Users/ketan/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift: eval: line 178: syntax error: unexpected end of file In summary, it seems impossible to pass single quote character (') to Swift commandline arg values. Thanks, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Feb 11 18:48:02 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 11 Feb 2014 16:48:02 -0800 Subject: [Swift-devel] some details about softimage In-Reply-To: <52FAC085.3000500@uchicago.edu> References: <1386713240.11830.3.camel@echo> <52FAC085.3000500@uchicago.edu> Message-ID: <1392166082.26419.1.camel@echo> On Tue, 2014-02-11 at 18:29 -0600, Yadu Nand B wrote: > Hi Mihael, > > Does this mechanism work with provider-staging and via passive > persistent coasters ? It's independent of how the coaster service is started, so yes. > Since you specify the , does this mechanism ensure that an > installation happens > only once per node ? Of course. It is, however, deleted when the last worker on a node quits. > Do other workers on the same node wait till the > first worker finishes > the installation ? Yes. > > I don't see this in the user-guide. Once I get this tested, I'll could > add a doc piece if that helps. It probably would help. Mihael From dsk at uchicago.edu Tue Feb 4 13:24:34 2014 From: dsk at uchicago.edu (Daniel Katz) Date: Tue, 04 Feb 2014 19:24:34 -0000 Subject: [Swift-devel] Migrating to github In-Reply-To: References: <85C85E44DD880E498CEA5A501B27954BEA4D27@DITKA.anl.gov> Message-ID: If moving to github, which seems like a good idea to me, why not also move from svn to git? On Tue, Feb 4, 2014 at 2:16 PM, Yadu Nand wrote: > I think moving to Github is a great idea, also makes sense to merge > cog and swift into one repo. > > Part of the reason for the popularity of Github is the social/open > aspect of development. Going > forward we'd definitely need that, and the sooner we migrate, the better. > > It also looks like there's some nice documentation here > http://git-scm.com/book/ch8-1.html > about git-svn interoperability. > > -Yadu > > On Tue, Feb 4, 2014 at 12:00 PM, Wilde, Michael J. > wrote: > > In in favor of considering this, and putting Swifts /K and /T in close > > proximity in the repo. > > > > I think its a step to becoming a more open open-source project. > > > > - Mike > > -- > > Michael Wilde > > Mathematics and Computer Science Computation Institute > > Argonne National Laboratory The University of Chicago > > > > ________________________________ > > From: swift-devel-bounces at ci.uchicago.edu > > [swift-devel-bounces at ci.uchicago.edu] on behalf of David Kelly > > [davidkelly at uchicago.edu] > > Sent: Tuesday, February 04, 2014 11:44 AM > > To: swift-devel > > Subject: [Swift-devel] Migrating to github > > > > Right now we have two repos - cog on sourceforge, and Swift on > > svn.ci.uchicago.edu. > > > > Sourceforge's SVN commit hooks are broken, so we do not get email updates > > automatically from them when a change is made. Instead I rigged a cron > job > > that runs every 5 minutes, checks for changes, and emails them out. It's > a > > little ugly. > > > > Developers wanting to make contributions to Swift need a CI account. This > > may discourage new developers from getting involved. > > > > Having two repos makes branching, tagging, and building more tedious > than it > > needs to be. > > > > This may need more discussion, but I would like to propose that starting > > with 0.95, let's merge cog and Swift and move to github. > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > -- > Yadu Nand B > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hereld at anl.gov Fri Feb 7 10:44:32 2014 From: hereld at anl.gov (Hereld, Mark) Date: Fri, 07 Feb 2014 16:44:32 -0000 Subject: [Swift-devel] Progress on Swift RAM usage problem? In-Reply-To: References: <85C85E44DD880E498CEA5A501B27954BEA237C@DITKA.anl.gov> <85C85E44DD880E498CEA5A501B27954BEA2486@DITKA.anl.gov> Message-ID: <999FFFA7-0555-48A7-A787-D0C9836DCE7C@anl.gov> nice plots! m On Feb 7, 2014, at 10:38 AM, David Kelly > wrote: For those interested in this problem, here is the latest heap plot of Jason's long (and still running) Beagle job. On Mon, Feb 3, 2014 at 3:29 AM, David Kelly > wrote: Hello, I've spent the weekend working on the popdiagts script. I looked around on Geyser's filesystem and was able to find some input files that I can use. Once I found the data and got the 39 arguments correct, I was able to reproduce the problem. I see a result that looks very similar to the initial report: Progress: time: Mon, 03 Feb 2014 01:20:00 -0700 Active:1 Finished successfully:3 /glade/u/home/davkelly/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift: line 177: 31567 Killed java -Xmx8096M -XX:+HeapDumpOnOutOfMemoryError -Djava.endorsed.dirs=/glade/u/home/davkelly/swift-0.94/cog/modules/s... To start, I ran Swift with the default of 1G heap size and within a few minutes I was able to see Swift being killed. A heap plot of a failing run: http://web.ci.uchicago.edu/~davidk/popdiagts-20140201-1458-i9hmaf0e.png I tried bumping up the max heap size, but I ran into the same problem within a few minutes. The amount of memory used never seems to get very high. Here is a plot with 8G: http://web.ci.uchicago.edu/~davidk/popdiagts-20140203-0059-g6a11m24.png I used jmap to generate several heap dumps during the run. They are about 100MB compressed, 400MB uncompressed, located at: http://web.ci.uchicago.edu/~davidk/heap1.gz http://web.ci.uchicago.edu/~davidk/heap2.gz http://web.ci.uchicago.edu/~davidk/heap3.gz http://web.ci.uchicago.edu/~davidk/heap4.gz http://web.ci.uchicago.edu/~davidk/heap5.gz http://web.ci.uchicago.edu/~davidk/heap6.gz http://web.ci.uchicago.edu/~davidk/heap7.gz http://web.ci.uchicago.edu/~davidk/heap8.gz http://web.ci.uchicago.edu/~davidk/heap9.gz http://web.ci.uchicago.edu/~davidk/heap10.gz http://web.ci.uchicago.edu/~davidk/heap11.gz I used Eclipse Memory Analyzer to look at the heaps. You can view an html histogram of the objects at: http://web.ci.uchicago.edu/~davidk/heap-histogram/index.html It's possible that there was a sudden spike in memory at the end that the logs missed, but I don't think that's what's going on here. As I was running the script, I opened top and saw the Swift CPU usage on the Geyser head node get extremely high, up to 700%. I think it's getting killed due to a kernel CPU throttle. I went through the script line by line until I could narrow down where the problem was. I whittling away at it until I could get a small, readable, and data-independent test script that shows the problem. Here it is: ---- type file; app (file out) createFile() { createFile @filename(out); } app (file out) createFileGivenArray (file fileArray[]) { createFile @filename(out); } file myArray[]; file myFile; foreach f,i in [1:2] { myArray[i] = createFile(); } myFile = createFileGivenArray(myArray); ----- On Midway you'll see the CPU usage on the head node jump to about 200% while the first app runs. If you repeat that pattern many times (like the original script does) you'll see CPU usage go even higher. I've filed this as Bug 1195 ( https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1195 ). The package to reproduce this is at http://web.ci.uchicago.edu/~davidk/popdiag.tar.gz. On Tue, Jan 28, 2014 at 3:29 PM, Wilde, Michael J. > wrote: From: David Kelly [davidkelly at uchicago.edu] Sent: Tuesday, January 28, 2014 2:47 PM ... I don't have too many updates on Sheri's problem. I was able to run the older standalone example I had on Geyser and did not see any issues with excessive amounts of resident memory being used. ... I think the failure was exceeding the Java heap size, not an RSS problem, right? I think we might be better off shifting the way we approach this problem. It's difficult to run these apps, and to run them in the same way the users do. There's also a long delay getting responses. I think we'd be better off focusing on adding comprehensive memory tests to the test suite, measuring, plotting, and then documenting solutions/strategies into the user guide. It will take some time, but I think it's the best approach since everything would be under our own control, and it would provide solutions for all users. That sounds good, while we are waiting for debugging info from users. But we should still strive to reproduce problems that users are encountering, and on giving them code updates with additional debugging hooks or possible remedies to test. - Mike On Tue, Jan 28, 2014 at 12:36 PM, Wilde, Michael J. > wrote: Yadu, David, can you send updates on this to Swift devel, and lets talk this afternoon at 3PM to discuss? Thanks, - Mike _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel ------------------------------------------------------- Mark Hereld > Senior Fellow - Computation Institute Experimental Systems Engineer - Mathematics and Computer Science Visualization and Analysis Lead - Argonne Leadership Computing Facility Argonne National Laboratory The University of Chicago Cell: 630.327.2088 Voice: 630.252.4170 -------------- next part -------------- An HTML attachment was scrubbed... URL: From yadunand at uchicago.edu Tue Feb 11 18:29:59 2014 From: yadunand at uchicago.edu (Yadu Nand B) Date: Wed, 12 Feb 2014 00:29:59 -0000 Subject: [Swift-devel] some details about softimage In-Reply-To: <1386713240.11830.3.camel@echo> References: <1386713240.11830.3.camel@echo> Message-ID: <52FAC085.3000500@uchicago.edu> Hi Mihael, Does this mechanism work with provider-staging and via passive persistent coasters ? Since you specify the , does this mechanism ensure that an installation happens only once per node ? Do other workers on the same node wait till the first worker finishes the installation ? I don't see this in the user-guide. Once I get this tested, I'll could add a doc piece if that helps. Thanks, Yadu On 12/10/2013 04:07 PM, Mihael Hategan wrote: > Mike asked for some details of how that works. > > You specify it like this in sites.xml: > > key="softImage">/lustre/beagle/hategan/softimg.tar.gz /tmp/softimage > > That is . > > If your tar.gz contains a "start" executable in the root directory, it > will be executed after unpacking. > > Similarly, if the tar.gz contains a "stop" executable in the root > directory, it will be executed before cleanup (which is when no more > jobs will be run on that node). > > Mihael > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From yadunand at uchicago.edu Wed Feb 12 13:54:16 2014 From: yadunand at uchicago.edu (Yadu Nand B) Date: Wed, 12 Feb 2014 19:54:16 -0000 Subject: [Swift-devel] High cpu usage bug and large logs Message-ID: <52FBD14E.5070703@uchicago.edu> Hi, In swift scripts which have an app waiting on an array closing such as the part06 in the CIC tutorial ( https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/CIC_2013-08-09/part06 ) Swift release candidates 0.94.2-RC2, 0.94.2-RC3 show following behavior: swift-0.94.1 swift-0.94.2-RC2 & RC3 cpu usage <70% >300% on the osgconnect headnode. Log size ~200Kb +90Mb With larger swift scripts this problem becomes worse. Mihael has fixes for this issue, which are in Swift-0.95, but haven't been merged into the 0.94.2 RCs. Setting tracing.enabled=false fixes the log aspect of the issue. Do we make a new release candidate with the fix for the high cpu usage ? High cpu usage bug : https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1195 Thanks, Yadu -------------- next part -------------- An HTML attachment was scrubbed... URL: