From ketan at mcs.anl.gov Tue Mar 3 14:01:28 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Tue, 3 Mar 2015 14:01:28 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely Message-ID: Hi, Continuing the discussion on devel. It seems that the run worked after I changed the staging method from "direct" to "swift". I am trying to narrow down the cause why "direct" staging does not work. Any pointers to possible causes will help. Thanks, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Mar 3 14:51:39 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 3 Mar 2015 12:51:39 -0800 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: Message-ID: <1425415899.22126.6.camel@echo> With direct "staging" and a slow network FS, the application run time will go up. This is why in many cases "avoid NFS/gpfs" is a good strategy. What happens if you increase the walltime for your jobs? Mihael On Tue, 2015-03-03 at 14:01 -0600, Ketan Maheshwari wrote: > Hi, > > Continuing the discussion on devel. It seems that the run worked after I > changed the staging method from "direct" to "swift". > > I am trying to narrow down the cause why "direct" staging does not work. > Any pointers to possible causes will help. > > Thanks, > Ketan > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From ketan at mcs.anl.gov Tue Mar 3 15:42:42 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Tue, 3 Mar 2015 15:42:42 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: Message-ID: Slow network looks unlikely to be a cause: I tried with 1 app call, total I/O size less than 20KB and a job wall-time of 40 minutes. I still see the hang. The output files produced by the app do end up in the outdir. Another observation is that despite 40 minutes of walltime, the application crashes in 2 minutes with a message saying walltime exceeded, as follows: exception @ swift-int-staging.k, line: 160 Caused by: Walltime exceeded k:assign @ swift.k, line: 174 Caused by: Exception in bgsh: Arguments: [/home/ketan/SwiftApps/subjobs/mpicatsnsleep/mpicatnap, /gpfs/mira-home/ketan/SwiftApps/subjobs/mpicatsnsleep/./data.txt, /gpfs/mira-home/ketan/SwiftApps/subjobs/mpicatsnsleep/./outdir/f.0001.out, 1] Host: cluster Directory: catsnsleepmpi-run001/jobs/b/bgsh-k7exhe5m exception @ swift-int-staging.k, line: 165 --Ketan On Tue, Mar 3, 2015 at 2:51 PM, Hategan-Marandiuc, Philip M. < hategan at mcs.anl.gov> wrote: > With direct "staging" and a slow network FS, the application run time > will go up. This is why in many cases "avoid NFS/gpfs" is a good > strategy. > > What happens if you increase the walltime for your jobs? > > Mihael > > On Tue, 2015-03-03 at 14:01 -0600, Ketan Maheshwari wrote: > > Hi, > > > > Continuing the discussion on devel. It seems that the run worked after I > > changed the staging method from "direct" to "swift". > > > > I am trying to narrow down the cause why "direct" staging does not work. > > Any pointers to possible causes will help. > > > > Thanks, > > Ketan > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Mar 3 19:03:27 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 3 Mar 2015 17:03:27 -0800 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: Message-ID: <1425431007.22126.9.camel@echo> On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > Slow network looks unlikely to be a cause: It's the only variable obvious, so I wouldn't say that. > > I tried with 1 app call, total I/O size less than 20KB and a job wall-time > of 40 minutes. I still see the hang. The output files produced by the app > do end up in the outdir. > > Another observation is that despite 40 minutes of walltime, the application > crashes in 2 minutes with a message saying walltime exceeded, as follows: Then the walltime wasn't increased. Can you, please, ALWAYS post the log when there is such an issue so we can troubleshoot things? Mihael From ketan at mcs.anl.gov Tue Mar 3 19:53:08 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Tue, 3 Mar 2015 19:53:08 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> Message-ID: Please find the log attached. --Ketan On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < hategan at mcs.anl.gov> wrote: > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > > Slow network looks unlikely to be a cause: > > It's the only variable obvious, so I wouldn't say that. > > > > > I tried with 1 app call, total I/O size less than 20KB and a job > wall-time > > of 40 minutes. I still see the hang. The output files produced by the app > > do end up in the outdir. > > > > Another observation is that despite 40 minutes of walltime, the > application > > crashes in 2 minutes with a message saying walltime exceeded, as follows: > > Then the walltime wasn't increased. Can you, please, ALWAYS post the log > when there is such an issue so we can troubleshoot things? > > Mihael > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: run001.tgz Type: application/x-gzip Size: 24250 bytes Desc: not available URL: From hategan at mcs.anl.gov Tue Mar 3 22:03:54 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 3 Mar 2015 20:03:54 -0800 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> Message-ID: <1425441834.22093.4.camel@echo> Hi, Looks like almost exactly 5 minutes to me: 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE taskid=urn:R-3-0-2-1425432781969 status=2 workerid=0304-3301040-000000:000000 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded Which is what the config file is asking for: app.bgsh { env.SUBBLOCK_SIZE: "16" # [R] line 27 executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] line 25 maxWallTime: "00:05:00" # [R] line 26 } Again, the wrapper log shows the app as still running. Last line is: Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE Please do me a favor and increase the walltime to one hour and let's see what happens then. If it still doesn't finish after one hour, we could try to strace it and see what is happening there. Mihael On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: > Please find the log attached. --Ketan > > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < > hategan at mcs.anl.gov> wrote: > > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > > > Slow network looks unlikely to be a cause: > > > > It's the only variable obvious, so I wouldn't say that. I meant "only obvious variable" there. From ketan at mcs.anl.gov Tue Mar 3 22:15:41 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Tue, 3 Mar 2015 22:15:41 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> Message-ID: When I check queue with qstat, I see the job is submitted for 40 minutes. When I try to increase maxWallTime the workflow does not get submitted because on Cetus maximum allowed walltime is 60 minutes. --Ketan On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < hategan at mcs.anl.gov> wrote: > Hi, > > Looks like almost exactly 5 minutes to me: > > 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE > taskid=urn:R-3-0-2-1425432781969 status=2 > workerid=0304-3301040-000000:000000 > 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE > taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded > > Which is what the config file is asking for: > > app.bgsh { > env.SUBBLOCK_SIZE: "16" # [R] line 27 > executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] line 25 > maxWallTime: "00:05:00" # [R] line 26 > } > > Again, the wrapper log shows the app as still running. Last line is: > Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE > > Please do me a favor and increase the walltime to one hour and let's see > what happens then. > > If it still doesn't finish after one hour, we could try to strace it and > see what is happening there. > > Mihael > > On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: > > Please find the log attached. --Ketan > > > > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < > > hategan at mcs.anl.gov> wrote: > > > > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > > > > Slow network looks unlikely to be a cause: > > > > > > It's the only variable obvious, so I wouldn't say that. > > I meant "only obvious variable" there. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Tue Mar 3 22:28:53 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Tue, 3 Mar 2015 22:28:53 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> Message-ID: Attached is a log for maxWalltime set to 7 minutes beyond which the job does not get submitted because of the 1 hour walltime limit of Cetus. --Ketan On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari wrote: > When I check queue with qstat, I see the job is submitted for 40 minutes. > When I try to increase maxWallTime the workflow does not get submitted > because on Cetus maximum allowed walltime is 60 minutes. --Ketan > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < > hategan at mcs.anl.gov> wrote: > >> Hi, >> >> Looks like almost exactly 5 minutes to me: >> >> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE >> taskid=urn:R-3-0-2-1425432781969 status=2 >> workerid=0304-3301040-000000:000000 >> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded >> >> Which is what the config file is asking for: >> >> app.bgsh { >> env.SUBBLOCK_SIZE: "16" # [R] line 27 >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] line 25 >> maxWallTime: "00:05:00" # [R] line 26 >> } >> >> Again, the wrapper log shows the app as still running. Last line is: >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE >> >> Please do me a favor and increase the walltime to one hour and let's see >> what happens then. >> >> If it still doesn't finish after one hour, we could try to strace it and >> see what is happening there. >> >> Mihael >> >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: >> > Please find the log attached. --Ketan >> > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < >> > hategan at mcs.anl.gov> wrote: >> > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: >> > > > Slow network looks unlikely to be a cause: >> > > >> > > It's the only variable obvious, so I wouldn't say that. >> >> I meant "only obvious variable" there. >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: run005.tgz Type: application/x-gzip Size: 18465 bytes Desc: not available URL: From hategan at mcs.anl.gov Tue Mar 3 23:17:29 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 3 Mar 2015 21:17:29 -0800 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> Message-ID: <1425446249.27766.2.camel@echo> You are using coasters, so what gets queued is the block, not the job. You should specify execution.options.maxJobTime = "00:59:00". Then you can probably do a walltime of about "00:50:00". But 7 minutes vs. 5 minutes isn't much of a difference. Mihael On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: > Attached is a log for maxWalltime set to 7 minutes beyond which the job > does not get submitted because of the 1 hour walltime limit of Cetus. > --Ketan > > On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari wrote: > > > When I check queue with qstat, I see the job is submitted for 40 minutes. > > When I try to increase maxWallTime the workflow does not get submitted > > because on Cetus maximum allowed walltime is 60 minutes. --Ketan > > > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < > > hategan at mcs.anl.gov> wrote: > > > >> Hi, > >> > >> Looks like almost exactly 5 minutes to me: > >> > >> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE > >> taskid=urn:R-3-0-2-1425432781969 status=2 > >> workerid=0304-3301040-000000:000000 > >> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE > >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded > >> > >> Which is what the config file is asking for: > >> > >> app.bgsh { > >> env.SUBBLOCK_SIZE: "16" # [R] line 27 > >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] line 25 > >> maxWallTime: "00:05:00" # [R] line 26 > >> } > >> > >> Again, the wrapper log shows the app as still running. Last line is: > >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE > >> > >> Please do me a favor and increase the walltime to one hour and let's see > >> what happens then. > >> > >> If it still doesn't finish after one hour, we could try to strace it and > >> see what is happening there. > >> > >> Mihael > >> > >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: > >> > Please find the log attached. --Ketan > >> > > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < > >> > hategan at mcs.anl.gov> wrote: > >> > > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > >> > > > Slow network looks unlikely to be a cause: > >> > > > >> > > It's the only variable obvious, so I wouldn't say that. > >> > >> I meant "only obvious variable" there. > >> > >> > >> > > From ketan at mcs.anl.gov Wed Mar 4 08:44:39 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Wed, 4 Mar 2015 08:44:39 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: <1425446249.27766.2.camel@echo> References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> <1425446249.27766.2.camel@echo> Message-ID: Please find one with 59 minutes attached. --Ketan On Tue, Mar 3, 2015 at 11:17 PM, Mihael Hategan wrote: > You are using coasters, so what gets queued is the block, not the job. > > You should specify execution.options.maxJobTime = "00:59:00". > > Then you can probably do a walltime of about "00:50:00". But 7 minutes > vs. 5 minutes isn't much of a difference. > > Mihael > > On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: > > Attached is a log for maxWalltime set to 7 minutes beyond which the job > > does not get submitted because of the 1 hour walltime limit of Cetus. > > --Ketan > > > > On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari > wrote: > > > > > When I check queue with qstat, I see the job is submitted for 40 > minutes. > > > When I try to increase maxWallTime the workflow does not get submitted > > > because on Cetus maximum allowed walltime is 60 minutes. --Ketan > > > > > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < > > > hategan at mcs.anl.gov> wrote: > > > > > >> Hi, > > >> > > >> Looks like almost exactly 5 minutes to me: > > >> > > >> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE > > >> taskid=urn:R-3-0-2-1425432781969 status=2 > > >> workerid=0304-3301040-000000:000000 > > >> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE > > >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded > > >> > > >> Which is what the config file is asking for: > > >> > > >> app.bgsh { > > >> env.SUBBLOCK_SIZE: "16" # [R] line > 27 > > >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] line > 25 > > >> maxWallTime: "00:05:00" # [R] line > 26 > > >> } > > >> > > >> Again, the wrapper log shows the app as still running. Last line is: > > >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE > > >> > > >> Please do me a favor and increase the walltime to one hour and let's > see > > >> what happens then. > > >> > > >> If it still doesn't finish after one hour, we could try to strace it > and > > >> see what is happening there. > > >> > > >> Mihael > > >> > > >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: > > >> > Please find the log attached. --Ketan > > >> > > > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < > > >> > hategan at mcs.anl.gov> wrote: > > >> > > > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > > >> > > > Slow network looks unlikely to be a cause: > > >> > > > > >> > > It's the only variable obvious, so I wouldn't say that. > > >> > > >> I meant "only obvious variable" there. > > >> > > >> > > >> > > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: run009.tgz Type: application/x-gzip Size: 73947 bytes Desc: not available URL: From ketan at mcs.anl.gov Wed Mar 4 08:50:38 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Wed, 4 Mar 2015 08:50:38 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> <1425446249.27766.2.camel@echo> Message-ID: I added stdin="/dev/null" to app invocation line and it has worked now. --Ketan On Wed, Mar 4, 2015 at 8:44 AM, Ketan Maheshwari wrote: > Please find one with 59 minutes attached. --Ketan > > On Tue, Mar 3, 2015 at 11:17 PM, Mihael Hategan > wrote: > >> You are using coasters, so what gets queued is the block, not the job. >> >> You should specify execution.options.maxJobTime = "00:59:00". >> >> Then you can probably do a walltime of about "00:50:00". But 7 minutes >> vs. 5 minutes isn't much of a difference. >> >> Mihael >> >> On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: >> > Attached is a log for maxWalltime set to 7 minutes beyond which the job >> > does not get submitted because of the 1 hour walltime limit of Cetus. >> > --Ketan >> > >> > On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari >> wrote: >> > >> > > When I check queue with qstat, I see the job is submitted for 40 >> minutes. >> > > When I try to increase maxWallTime the workflow does not get submitted >> > > because on Cetus maximum allowed walltime is 60 minutes. --Ketan >> > > >> > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < >> > > hategan at mcs.anl.gov> wrote: >> > > >> > >> Hi, >> > >> >> > >> Looks like almost exactly 5 minutes to me: >> > >> >> > >> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE >> > >> taskid=urn:R-3-0-2-1425432781969 status=2 >> > >> workerid=0304-3301040-000000:000000 >> > >> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE >> > >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded >> > >> >> > >> Which is what the config file is asking for: >> > >> >> > >> app.bgsh { >> > >> env.SUBBLOCK_SIZE: "16" # [R] line >> 27 >> > >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] line >> 25 >> > >> maxWallTime: "00:05:00" # [R] line >> 26 >> > >> } >> > >> >> > >> Again, the wrapper log shows the app as still running. Last line is: >> > >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE >> > >> >> > >> Please do me a favor and increase the walltime to one hour and let's >> see >> > >> what happens then. >> > >> >> > >> If it still doesn't finish after one hour, we could try to strace it >> and >> > >> see what is happening there. >> > >> >> > >> Mihael >> > >> >> > >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: >> > >> > Please find the log attached. --Ketan >> > >> > >> > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < >> > >> > hategan at mcs.anl.gov> wrote: >> > >> > >> > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: >> > >> > > > Slow network looks unlikely to be a cause: >> > >> > > >> > >> > > It's the only variable obvious, so I wouldn't say that. >> > >> >> > >> I meant "only obvious variable" there. >> > >> >> > >> >> > >> >> > > >> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Wed Mar 4 15:12:28 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 4 Mar 2015 13:12:28 -0800 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> <1425446249.27766.2.camel@echo> Message-ID: <1425503548.30155.4.camel@echo> I'm still confused. I don't see any difference in stdin handling between _swiftwrap and _swiftwrap.staging (which is used for direct staging). Maybe we should always feed the app a /dev/null if there is no stdin= specified. Mihael On Wed, 2015-03-04 at 08:50 -0600, Ketan Maheshwari wrote: > I added stdin="/dev/null" to app invocation line and it has worked now. > --Ketan > > On Wed, Mar 4, 2015 at 8:44 AM, Ketan Maheshwari wrote: > > > Please find one with 59 minutes attached. --Ketan > > > > On Tue, Mar 3, 2015 at 11:17 PM, Mihael Hategan > > wrote: > > > >> You are using coasters, so what gets queued is the block, not the job. > >> > >> You should specify execution.options.maxJobTime = "00:59:00". > >> > >> Then you can probably do a walltime of about "00:50:00". But 7 minutes > >> vs. 5 minutes isn't much of a difference. > >> > >> Mihael > >> > >> On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: > >> > Attached is a log for maxWalltime set to 7 minutes beyond which the job > >> > does not get submitted because of the 1 hour walltime limit of Cetus. > >> > --Ketan > >> > > >> > On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari > >> wrote: > >> > > >> > > When I check queue with qstat, I see the job is submitted for 40 > >> minutes. > >> > > When I try to increase maxWallTime the workflow does not get submitted > >> > > because on Cetus maximum allowed walltime is 60 minutes. --Ketan > >> > > > >> > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < > >> > > hategan at mcs.anl.gov> wrote: > >> > > > >> > >> Hi, > >> > >> > >> > >> Looks like almost exactly 5 minutes to me: > >> > >> > >> > >> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE > >> > >> taskid=urn:R-3-0-2-1425432781969 status=2 > >> > >> workerid=0304-3301040-000000:000000 > >> > >> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE > >> > >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded > >> > >> > >> > >> Which is what the config file is asking for: > >> > >> > >> > >> app.bgsh { > >> > >> env.SUBBLOCK_SIZE: "16" # [R] line > >> 27 > >> > >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] line > >> 25 > >> > >> maxWallTime: "00:05:00" # [R] line > >> 26 > >> > >> } > >> > >> > >> > >> Again, the wrapper log shows the app as still running. Last line is: > >> > >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE > >> > >> > >> > >> Please do me a favor and increase the walltime to one hour and let's > >> see > >> > >> what happens then. > >> > >> > >> > >> If it still doesn't finish after one hour, we could try to strace it > >> and > >> > >> see what is happening there. > >> > >> > >> > >> Mihael > >> > >> > >> > >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: > >> > >> > Please find the log attached. --Ketan > >> > >> > > >> > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < > >> > >> > hategan at mcs.anl.gov> wrote: > >> > >> > > >> > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > >> > >> > > > Slow network looks unlikely to be a cause: > >> > >> > > > >> > >> > > It's the only variable obvious, so I wouldn't say that. > >> > >> > >> > >> I meant "only obvious variable" there. > >> > >> > >> > >> > >> > >> > >> > > > >> > >> > >> _______________________________________________ > >> Swift-devel mailing list > >> Swift-devel at ci.uchicago.edu > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > >> > > > > From wilde at anl.gov Wed Mar 4 15:17:26 2015 From: wilde at anl.gov (Michael Wilde) Date: Wed, 4 Mar 2015 15:17:26 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: <1425503548.30155.4.camel@echo> References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> <1425446249.27766.2.camel@echo> <1425503548.30155.4.camel@echo> Message-ID: <54F77666.4000607@anl.gov> This brings to mind a similar problem that we encountered a while back, with an MPICH2 bug. See this ticket: https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1014 - Mike On 3/4/15 3:12 PM, Mihael Hategan wrote: > I'm still confused. I don't see any difference in stdin handling between > _swiftwrap and _swiftwrap.staging (which is used for direct staging). > > Maybe we should always feed the app a /dev/null if there is no stdin= > specified. > > Mihael > > On Wed, 2015-03-04 at 08:50 -0600, Ketan Maheshwari wrote: >> I added stdin="/dev/null" to app invocation line and it has worked now. >> --Ketan >> >> On Wed, Mar 4, 2015 at 8:44 AM, Ketan Maheshwari wrote: >> >>> Please find one with 59 minutes attached. --Ketan >>> >>> On Tue, Mar 3, 2015 at 11:17 PM, Mihael Hategan >>> wrote: >>> >>>> You are using coasters, so what gets queued is the block, not the job. >>>> >>>> You should specify execution.options.maxJobTime = "00:59:00". >>>> >>>> Then you can probably do a walltime of about "00:50:00". But 7 minutes >>>> vs. 5 minutes isn't much of a difference. >>>> >>>> Mihael >>>> >>>> On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: >>>>> Attached is a log for maxWalltime set to 7 minutes beyond which the job >>>>> does not get submitted because of the 1 hour walltime limit of Cetus. >>>>> --Ketan >>>>> >>>>> On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari >>>> wrote: >>>>>> When I check queue with qstat, I see the job is submitted for 40 >>>> minutes. >>>>>> When I try to increase maxWallTime the workflow does not get submitted >>>>>> because on Cetus maximum allowed walltime is 60 minutes. --Ketan >>>>>> >>>>>> On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < >>>>>> hategan at mcs.anl.gov> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Looks like almost exactly 5 minutes to me: >>>>>>> >>>>>>> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE >>>>>>> taskid=urn:R-3-0-2-1425432781969 status=2 >>>>>>> workerid=0304-3301040-000000:000000 >>>>>>> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE >>>>>>> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded >>>>>>> >>>>>>> Which is what the config file is asking for: >>>>>>> >>>>>>> app.bgsh { >>>>>>> env.SUBBLOCK_SIZE: "16" # [R] line >>>> 27 >>>>>>> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] line >>>> 25 >>>>>>> maxWallTime: "00:05:00" # [R] line >>>> 26 >>>>>>> } >>>>>>> >>>>>>> Again, the wrapper log shows the app as still running. Last line is: >>>>>>> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE >>>>>>> >>>>>>> Please do me a favor and increase the walltime to one hour and let's >>>> see >>>>>>> what happens then. >>>>>>> >>>>>>> If it still doesn't finish after one hour, we could try to strace it >>>> and >>>>>>> see what is happening there. >>>>>>> >>>>>>> Mihael >>>>>>> >>>>>>> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: >>>>>>>> Please find the log attached. --Ketan >>>>>>>> >>>>>>>> On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < >>>>>>>> hategan at mcs.anl.gov> wrote: >>>>>>>> >>>>>>>>> On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: >>>>>>>>>> Slow network looks unlikely to be a cause: >>>>>>>>> It's the only variable obvious, so I wouldn't say that. >>>>>>> I meant "only obvious variable" there. >>>>>>> >>>>>>> >>>>>>> >>>> >>>> _______________________________________________ >>>> Swift-devel mailing list >>>> Swift-devel at ci.uchicago.edu >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>> >>> > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From ketan at mcs.anl.gov Wed Mar 4 15:22:56 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Wed, 4 Mar 2015 15:22:56 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: <718607ec8b92474099f25b47736a6750@GEORGE.anl.gov> References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> <1425446249.27766.2.camel@echo> <718607ec8b92474099f25b47736a6750@GEORGE.anl.gov> Message-ID: Hi Mihael, The code in _swiftwrap.staging branches based on if STDIN is present or not. See the snippet below: if [ "$STDIN" == "" ]; then if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then echo "#!/bin/bash" > run.sh echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\"" >> run.sh chmod +x run.sh fi "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" else if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then echo "#!/bin/bash" > run.sh echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\" <\"$STDIN\"" >> run.sh chmod +x run.sh fi "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" fi When "stdin=" is not provided the code takes the first branch and hangs. It works otherwise. It is possible that it hangs because if mpich bug Mike mentioned. I agree we should stick in a wrote: > I'm still confused. I don't see any difference in stdin handling between > _swiftwrap and _swiftwrap.staging (which is used for direct staging). > > Maybe we should always feed the app a /dev/null if there is no stdin= > specified. > > Mihael > > On Wed, 2015-03-04 at 08:50 -0600, Ketan Maheshwari wrote: > > I added stdin="/dev/null" to app invocation line and it has worked now. > > --Ketan > > > > On Wed, Mar 4, 2015 at 8:44 AM, Ketan Maheshwari > wrote: > > > > > Please find one with 59 minutes attached. --Ketan > > > > > > On Tue, Mar 3, 2015 at 11:17 PM, Mihael Hategan > > > wrote: > > > > > >> You are using coasters, so what gets queued is the block, not the job. > > >> > > >> You should specify execution.options.maxJobTime = "00:59:00". > > >> > > >> Then you can probably do a walltime of about "00:50:00". But 7 minutes > > >> vs. 5 minutes isn't much of a difference. > > >> > > >> Mihael > > >> > > >> On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: > > >> > Attached is a log for maxWalltime set to 7 minutes beyond which the > job > > >> > does not get submitted because of the 1 hour walltime limit of > Cetus. > > >> > --Ketan > > >> > > > >> > On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari < > ketan at mcs.anl.gov> > > >> wrote: > > >> > > > >> > > When I check queue with qstat, I see the job is submitted for 40 > > >> minutes. > > >> > > When I try to increase maxWallTime the workflow does not get > submitted > > >> > > because on Cetus maximum allowed walltime is 60 minutes. --Ketan > > >> > > > > >> > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < > > >> > > hategan at mcs.anl.gov> wrote: > > >> > > > > >> > >> Hi, > > >> > >> > > >> > >> Looks like almost exactly 5 minutes to me: > > >> > >> > > >> > >> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE > > >> > >> taskid=urn:R-3-0-2-1425432781969 status=2 > > >> > >> workerid=0304-3301040-000000:000000 > > >> > >> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE > > >> > >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded > > >> > >> > > >> > >> Which is what the config file is asking for: > > >> > >> > > >> > >> app.bgsh { > > >> > >> env.SUBBLOCK_SIZE: "16" # [R] > line > > >> 27 > > >> > >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] > line > > >> 25 > > >> > >> maxWallTime: "00:05:00" # [R] > line > > >> 26 > > >> > >> } > > >> > >> > > >> > >> Again, the wrapper log shows the app as still running. Last line > is: > > >> > >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE > > >> > >> > > >> > >> Please do me a favor and increase the walltime to one hour and > let's > > >> see > > >> > >> what happens then. > > >> > >> > > >> > >> If it still doesn't finish after one hour, we could try to > strace it > > >> and > > >> > >> see what is happening there. > > >> > >> > > >> > >> Mihael > > >> > >> > > >> > >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: > > >> > >> > Please find the log attached. --Ketan > > >> > >> > > > >> > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < > > >> > >> > hategan at mcs.anl.gov> wrote: > > >> > >> > > > >> > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > > >> > >> > > > Slow network looks unlikely to be a cause: > > >> > >> > > > > >> > >> > > It's the only variable obvious, so I wouldn't say that. > > >> > >> > > >> > >> I meant "only obvious variable" there. > > >> > >> > > >> > >> > > >> > >> > > >> > > > > >> > > >> > > >> _______________________________________________ > > >> Swift-devel mailing list > > >> Swift-devel at ci.uchicago.edu > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > >> > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Wed Mar 4 15:37:18 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 4 Mar 2015 13:37:18 -0800 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> <1425446249.27766.2.camel@echo> <718607ec8b92474099f25b47736a6750@GEORGE.anl.gov> Message-ID: <1425505038.32265.4.camel@echo> Right. However, you mentioned before that it works fine when you switch staging from "direct" to "swift". That causes swift to use the plain _swiftwrap, which pretty much has the same code: if [ "$STDIN" == "" ]; then if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi if [ -n "$TIMECMD" ] && [ -n "$TIMEARGS" ]; then "$TIMECMD" "${TIMEARGS[@]}" "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" else "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" fi else ... So that's what's puzzling to me. Can you try changing this line in _swiftwrap.staging: "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" to "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" Hi Mihael, > > The code in _swiftwrap.staging branches based on if STDIN is present or > not. See the snippet below: > > if [ "$STDIN" == "" ]; then > if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then > echo "#!/bin/bash" > run.sh > echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\"" >> > run.sh > chmod +x run.sh > fi > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" > else > if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then > echo "#!/bin/bash" > run.sh > echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\" > <\"$STDIN\"" >> run.sh > chmod +x run.sh > fi > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" > fi > > When "stdin=" is not provided the code takes the first branch and hangs. It > works otherwise. > > It is possible that it hangs because if mpich bug Mike mentioned. > > I agree we should stick in a > --Ketan > > > On Wed, Mar 4, 2015 at 3:12 PM, Hategan-Marandiuc, Philip M. < > hategan at mcs.anl.gov> wrote: > > > I'm still confused. I don't see any difference in stdin handling between > > _swiftwrap and _swiftwrap.staging (which is used for direct staging). > > > > Maybe we should always feed the app a /dev/null if there is no stdin= > > specified. > > > > Mihael > > > > On Wed, 2015-03-04 at 08:50 -0600, Ketan Maheshwari wrote: > > > I added stdin="/dev/null" to app invocation line and it has worked now. > > > --Ketan > > > > > > On Wed, Mar 4, 2015 at 8:44 AM, Ketan Maheshwari > > wrote: > > > > > > > Please find one with 59 minutes attached. --Ketan > > > > > > > > On Tue, Mar 3, 2015 at 11:17 PM, Mihael Hategan > > > > wrote: > > > > > > > >> You are using coasters, so what gets queued is the block, not the job. > > > >> > > > >> You should specify execution.options.maxJobTime = "00:59:00". > > > >> > > > >> Then you can probably do a walltime of about "00:50:00". But 7 minutes > > > >> vs. 5 minutes isn't much of a difference. > > > >> > > > >> Mihael > > > >> > > > >> On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: > > > >> > Attached is a log for maxWalltime set to 7 minutes beyond which the > > job > > > >> > does not get submitted because of the 1 hour walltime limit of > > Cetus. > > > >> > --Ketan > > > >> > > > > >> > On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari < > > ketan at mcs.anl.gov> > > > >> wrote: > > > >> > > > > >> > > When I check queue with qstat, I see the job is submitted for 40 > > > >> minutes. > > > >> > > When I try to increase maxWallTime the workflow does not get > > submitted > > > >> > > because on Cetus maximum allowed walltime is 60 minutes. --Ketan > > > >> > > > > > >> > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. < > > > >> > > hategan at mcs.anl.gov> wrote: > > > >> > > > > > >> > >> Hi, > > > >> > >> > > > >> > >> Looks like almost exactly 5 minutes to me: > > > >> > >> > > > >> > >> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE > > > >> > >> taskid=urn:R-3-0-2-1425432781969 status=2 > > > >> > >> workerid=0304-3301040-000000:000000 > > > >> > >> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE > > > >> > >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded > > > >> > >> > > > >> > >> Which is what the config file is asking for: > > > >> > >> > > > >> > >> app.bgsh { > > > >> > >> env.SUBBLOCK_SIZE: "16" # [R] > > line > > > >> 27 > > > >> > >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # [R] > > line > > > >> 25 > > > >> > >> maxWallTime: "00:05:00" # [R] > > line > > > >> 26 > > > >> > >> } > > > >> > >> > > > >> > >> Again, the wrapper log shows the app as still running. Last line > > is: > > > >> > >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE > > > >> > >> > > > >> > >> Please do me a favor and increase the walltime to one hour and > > let's > > > >> see > > > >> > >> what happens then. > > > >> > >> > > > >> > >> If it still doesn't finish after one hour, we could try to > > strace it > > > >> and > > > >> > >> see what is happening there. > > > >> > >> > > > >> > >> Mihael > > > >> > >> > > > >> > >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: > > > >> > >> > Please find the log attached. --Ketan > > > >> > >> > > > > >> > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip M. < > > > >> > >> > hategan at mcs.anl.gov> wrote: > > > >> > >> > > > > >> > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari wrote: > > > >> > >> > > > Slow network looks unlikely to be a cause: > > > >> > >> > > > > > >> > >> > > It's the only variable obvious, so I wouldn't say that. > > > >> > >> > > > >> > >> I meant "only obvious variable" there. > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > > > > > >> > > > >> > > > >> _______________________________________________ > > > >> Swift-devel mailing list > > > >> Swift-devel at ci.uchicago.edu > > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > >> > > > > > > > > > > > > > > From ketan at mcs.anl.gov Wed Mar 4 15:43:38 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Wed, 4 Mar 2015 15:43:38 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: <1425505038.32265.4.camel@echo> References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> <1425446249.27766.2.camel@echo> <718607ec8b92474099f25b47736a6750@GEORGE.anl.gov> <1425505038.32265.4.camel@echo> Message-ID: Sorry, I realized your question now. I have copied the whole if ... fi block (including genScripts) from _swiftwrap to _swiftwrap.staging to see what happens. Will keep you posted. --Ketan On Wed, Mar 4, 2015 at 3:37 PM, Mihael Hategan wrote: > Right. However, you mentioned before that it works fine when you switch > staging from "direct" to "swift". That causes swift to use the plain > _swiftwrap, which pretty much has the same code: > > if [ "$STDIN" == "" ]; then > if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then > genScripts > fi > > if [ -n "$TIMECMD" ] && [ -n "$TIMEARGS" ]; then > "$TIMECMD" "${TIMEARGS[@]}" "$EXEC" "${CMDARGS[@]}" > 1>"$STDOUT" 2>"$STDERR" > else > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" > fi > else > ... > > So that's what's puzzling to me. > > Can you try changing this line in _swiftwrap.staging: > > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" > > to > > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" > and then remove stdin= from the app in the swift script and seeing if > that works? > > While it seems like a wise choice to do this in general, I'm trying to > see if this fixes this particular issue. > > Mihael > > > On Wed, 2015-03-04 at 15:22 -0600, Ketan Maheshwari wrote: > > Hi Mihael, > > > > The code in _swiftwrap.staging branches based on if STDIN is present or > > not. See the snippet below: > > > > if [ "$STDIN" == "" ]; then > > if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then > > echo "#!/bin/bash" > run.sh > > echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\"" >> > > run.sh > > chmod +x run.sh > > fi > > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" > > else > > if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then > > echo "#!/bin/bash" > run.sh > > echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\" > > <\"$STDIN\"" >> run.sh > > chmod +x run.sh > > fi > > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" > > fi > > > > When "stdin=" is not provided the code takes the first branch and hangs. > It > > works otherwise. > > > > It is possible that it hangs because if mpich bug Mike mentioned. > > > > I agree we should stick in a > > > --Ketan > > > > > > On Wed, Mar 4, 2015 at 3:12 PM, Hategan-Marandiuc, Philip M. < > > hategan at mcs.anl.gov> wrote: > > > > > I'm still confused. I don't see any difference in stdin handling > between > > > _swiftwrap and _swiftwrap.staging (which is used for direct staging). > > > > > > Maybe we should always feed the app a /dev/null if there is no stdin= > > > specified. > > > > > > Mihael > > > > > > On Wed, 2015-03-04 at 08:50 -0600, Ketan Maheshwari wrote: > > > > I added stdin="/dev/null" to app invocation line and it has worked > now. > > > > --Ketan > > > > > > > > On Wed, Mar 4, 2015 at 8:44 AM, Ketan Maheshwari > > > wrote: > > > > > > > > > Please find one with 59 minutes attached. --Ketan > > > > > > > > > > On Tue, Mar 3, 2015 at 11:17 PM, Mihael Hategan < > hategan at mcs.anl.gov> > > > > > wrote: > > > > > > > > > >> You are using coasters, so what gets queued is the block, not the > job. > > > > >> > > > > >> You should specify execution.options.maxJobTime = "00:59:00". > > > > >> > > > > >> Then you can probably do a walltime of about "00:50:00". But 7 > minutes > > > > >> vs. 5 minutes isn't much of a difference. > > > > >> > > > > >> Mihael > > > > >> > > > > >> On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: > > > > >> > Attached is a log for maxWalltime set to 7 minutes beyond which > the > > > job > > > > >> > does not get submitted because of the 1 hour walltime limit of > > > Cetus. > > > > >> > --Ketan > > > > >> > > > > > >> > On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari < > > > ketan at mcs.anl.gov> > > > > >> wrote: > > > > >> > > > > > >> > > When I check queue with qstat, I see the job is submitted for > 40 > > > > >> minutes. > > > > >> > > When I try to increase maxWallTime the workflow does not get > > > submitted > > > > >> > > because on Cetus maximum allowed walltime is 60 minutes. > --Ketan > > > > >> > > > > > > >> > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip M. > < > > > > >> > > hategan at mcs.anl.gov> wrote: > > > > >> > > > > > > >> > >> Hi, > > > > >> > >> > > > > >> > >> Looks like almost exactly 5 minutes to me: > > > > >> > >> > > > > >> > >> 2015-03-04 01:45:43,943+0000 INFO Execute TASK_STATUS_CHANGE > > > > >> > >> taskid=urn:R-3-0-2-1425432781969 status=2 > > > > >> > >> workerid=0304-3301040-000000:000000 > > > > >> > >> 2015-03-04 01:50:44,676+0000 INFO Execute TASK_STATUS_CHANGE > > > > >> > >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded > > > > >> > >> > > > > >> > >> Which is what the config file is asking for: > > > > >> > >> > > > > >> > >> app.bgsh { > > > > >> > >> env.SUBBLOCK_SIZE: "16" # > [R] > > > line > > > > >> 27 > > > > >> > >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # > [R] > > > line > > > > >> 25 > > > > >> > >> maxWallTime: "00:05:00" # > [R] > > > line > > > > >> 26 > > > > >> > >> } > > > > >> > >> > > > > >> > >> Again, the wrapper log shows the app as still running. Last > line > > > is: > > > > >> > >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE > > > > >> > >> > > > > >> > >> Please do me a favor and increase the walltime to one hour > and > > > let's > > > > >> see > > > > >> > >> what happens then. > > > > >> > >> > > > > >> > >> If it still doesn't finish after one hour, we could try to > > > strace it > > > > >> and > > > > >> > >> see what is happening there. > > > > >> > >> > > > > >> > >> Mihael > > > > >> > >> > > > > >> > >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: > > > > >> > >> > Please find the log attached. --Ketan > > > > >> > >> > > > > > >> > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip > M. < > > > > >> > >> > hategan at mcs.anl.gov> wrote: > > > > >> > >> > > > > > >> > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari > wrote: > > > > >> > >> > > > Slow network looks unlikely to be a cause: > > > > >> > >> > > > > > > >> > >> > > It's the only variable obvious, so I wouldn't say that. > > > > >> > >> > > > > >> > >> I meant "only obvious variable" there. > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > > > > > > >> > > > > >> > > > > >> _______________________________________________ > > > > >> Swift-devel mailing list > > > > >> Swift-devel at ci.uchicago.edu > > > > >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > >> > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Thu Mar 5 10:20:10 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Thu, 5 Mar 2015 10:20:10 -0600 Subject: [Swift-devel] trunk-cobalt block task ended prematurely In-Reply-To: References: <21e0160793d8413f97e6ee09ce2f660b@GEORGE.anl.gov> <2b9fba0ebff4437b9fb99bfbfb1397cf@LUCKMAN.anl.gov> <1425446249.27766.2.camel@echo> <718607ec8b92474099f25b47736a6750@GEORGE.anl.gov> <1425505038.32265.4.camel@echo> Message-ID: Hi Mihael, Moving the STDIN block from _swiftwrap to _swiftwrap.staging did not make any difference, ie. it did not work. Changing "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" to "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" wrote: > Sorry, I realized your question now. I have copied the whole if ... fi > block (including genScripts) from _swiftwrap to _swiftwrap.staging to see > what happens. Will keep you posted. --Ketan > > On Wed, Mar 4, 2015 at 3:37 PM, Mihael Hategan > wrote: > >> Right. However, you mentioned before that it works fine when you switch >> staging from "direct" to "swift". That causes swift to use the plain >> _swiftwrap, which pretty much has the same code: >> >> if [ "$STDIN" == "" ]; then >> if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then >> genScripts >> fi >> >> if [ -n "$TIMECMD" ] && [ -n "$TIMEARGS" ]; then >> "$TIMECMD" "${TIMEARGS[@]}" "$EXEC" "${CMDARGS[@]}" >> 1>"$STDOUT" 2>"$STDERR" >> else >> "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" >> fi >> else >> ... >> >> So that's what's puzzling to me. >> >> Can you try changing this line in _swiftwrap.staging: >> >> "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" >> >> to >> >> "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" > >> and then remove stdin= from the app in the swift script and seeing if >> that works? >> >> While it seems like a wise choice to do this in general, I'm trying to >> see if this fixes this particular issue. >> >> Mihael >> >> >> On Wed, 2015-03-04 at 15:22 -0600, Ketan Maheshwari wrote: >> > Hi Mihael, >> > >> > The code in _swiftwrap.staging branches based on if STDIN is present or >> > not. See the snippet below: >> > >> > if [ "$STDIN" == "" ]; then >> > if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then >> > echo "#!/bin/bash" > run.sh >> > echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\"" >> >> >> > run.sh >> > chmod +x run.sh >> > fi >> > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" >> > else >> > if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then >> > echo "#!/bin/bash" > run.sh >> > echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\" >> > <\"$STDIN\"" >> run.sh >> > chmod +x run.sh >> > fi >> > "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" >> > fi >> > >> > When "stdin=" is not provided the code takes the first branch and >> hangs. It >> > works otherwise. >> > >> > It is possible that it hangs because if mpich bug Mike mentioned. >> > >> > I agree we should stick in a > > >> > --Ketan >> > >> > >> > On Wed, Mar 4, 2015 at 3:12 PM, Hategan-Marandiuc, Philip M. < >> > hategan at mcs.anl.gov> wrote: >> > >> > > I'm still confused. I don't see any difference in stdin handling >> between >> > > _swiftwrap and _swiftwrap.staging (which is used for direct staging). >> > > >> > > Maybe we should always feed the app a /dev/null if there is no stdin= >> > > specified. >> > > >> > > Mihael >> > > >> > > On Wed, 2015-03-04 at 08:50 -0600, Ketan Maheshwari wrote: >> > > > I added stdin="/dev/null" to app invocation line and it has worked >> now. >> > > > --Ketan >> > > > >> > > > On Wed, Mar 4, 2015 at 8:44 AM, Ketan Maheshwari > > >> > > wrote: >> > > > >> > > > > Please find one with 59 minutes attached. --Ketan >> > > > > >> > > > > On Tue, Mar 3, 2015 at 11:17 PM, Mihael Hategan < >> hategan at mcs.anl.gov> >> > > > > wrote: >> > > > > >> > > > >> You are using coasters, so what gets queued is the block, not >> the job. >> > > > >> >> > > > >> You should specify execution.options.maxJobTime = "00:59:00". >> > > > >> >> > > > >> Then you can probably do a walltime of about "00:50:00". But 7 >> minutes >> > > > >> vs. 5 minutes isn't much of a difference. >> > > > >> >> > > > >> Mihael >> > > > >> >> > > > >> On Tue, 2015-03-03 at 22:28 -0600, Ketan Maheshwari wrote: >> > > > >> > Attached is a log for maxWalltime set to 7 minutes beyond >> which the >> > > job >> > > > >> > does not get submitted because of the 1 hour walltime limit of >> > > Cetus. >> > > > >> > --Ketan >> > > > >> > >> > > > >> > On Tue, Mar 3, 2015 at 10:15 PM, Ketan Maheshwari < >> > > ketan at mcs.anl.gov> >> > > > >> wrote: >> > > > >> > >> > > > >> > > When I check queue with qstat, I see the job is submitted >> for 40 >> > > > >> minutes. >> > > > >> > > When I try to increase maxWallTime the workflow does not get >> > > submitted >> > > > >> > > because on Cetus maximum allowed walltime is 60 minutes. >> --Ketan >> > > > >> > > >> > > > >> > > On Tue, Mar 3, 2015 at 10:03 PM, Hategan-Marandiuc, Philip >> M. < >> > > > >> > > hategan at mcs.anl.gov> wrote: >> > > > >> > > >> > > > >> > >> Hi, >> > > > >> > >> >> > > > >> > >> Looks like almost exactly 5 minutes to me: >> > > > >> > >> >> > > > >> > >> 2015-03-04 01:45:43,943+0000 INFO Execute >> TASK_STATUS_CHANGE >> > > > >> > >> taskid=urn:R-3-0-2-1425432781969 status=2 >> > > > >> > >> workerid=0304-3301040-000000:000000 >> > > > >> > >> 2015-03-04 01:50:44,676+0000 INFO Execute >> TASK_STATUS_CHANGE >> > > > >> > >> taskid=urn:R-3-0-2-1425432781969 status=5 Walltime exceeded >> > > > >> > >> >> > > > >> > >> Which is what the config file is asking for: >> > > > >> > >> >> > > > >> > >> app.bgsh { >> > > > >> > >> env.SUBBLOCK_SIZE: "16" # >> [R] >> > > line >> > > > >> 27 >> > > > >> > >> executable: "/home/ketan/SwiftApps/subjobs/bg.sh" # >> [R] >> > > line >> > > > >> 25 >> > > > >> > >> maxWallTime: "00:05:00" # >> [R] >> > > line >> > > > >> 26 >> > > > >> > >> } >> > > > >> > >> >> > > > >> > >> Again, the wrapper log shows the app as still running. Last >> line >> > > is: >> > > > >> > >> Progress 2015-03-04 01:45:43.971393118+0000 EXECUTE >> > > > >> > >> >> > > > >> > >> Please do me a favor and increase the walltime to one hour >> and >> > > let's >> > > > >> see >> > > > >> > >> what happens then. >> > > > >> > >> >> > > > >> > >> If it still doesn't finish after one hour, we could try to >> > > strace it >> > > > >> and >> > > > >> > >> see what is happening there. >> > > > >> > >> >> > > > >> > >> Mihael >> > > > >> > >> >> > > > >> > >> On Tue, 2015-03-03 at 19:53 -0600, Ketan Maheshwari wrote: >> > > > >> > >> > Please find the log attached. --Ketan >> > > > >> > >> > >> > > > >> > >> > On Tue, Mar 3, 2015 at 7:03 PM, Hategan-Marandiuc, Philip >> M. < >> > > > >> > >> > hategan at mcs.anl.gov> wrote: >> > > > >> > >> > >> > > > >> > >> > > On Tue, 2015-03-03 at 15:42 -0600, Ketan Maheshwari >> wrote: >> > > > >> > >> > > > Slow network looks unlikely to be a cause: >> > > > >> > >> > > >> > > > >> > >> > > It's the only variable obvious, so I wouldn't say that. >> > > > >> > >> >> > > > >> > >> I meant "only obvious variable" there. >> > > > >> > >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > > >> > > > >> >> > > > >> >> > > > >> _______________________________________________ >> > > > >> Swift-devel mailing list >> > > > >> Swift-devel at ci.uchicago.edu >> > > > >> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> > > > >> >> > > > > >> > > > > >> > > >> > > >> > > >> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Sat Mar 7 13:02:50 2015 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sat, 07 Mar 2015 13:02:50 -0600 Subject: [Swift-devel] CFP: IEEE Cluster 2015 -- Submission deadline extended to March 25th AOE! (hard deadline) Message-ID: <54FB4B5A.7000501@cs.iit.edu> March 7, 2015 Release IEEE International Conference on Cluster Computing September 8-11, 2015 Chicago, IL, USA http://www.mcs.anl.gov/ieeecluster2015/ CLUSTER 2015 CALL FOR PAPERS Following the successes of the series of Cluster conferences, for 2015 we solicit high-quality original papers presenting work that advances the state-of-the-art in clusters and closely related fields. All papers will be rigorously peer-reviewed for their originality, technical depth and correctness, potential impact, relevance to the conference, and quality of presentation. Research papers must clearly demonstrate research contributions and novelty, while papers reporting experience must clearly describe lessons learned and impact, along with the utility of the approach compared to the ones in the past. *** Paper Tracks *** Area 1: Application, Algorithms, and Libraries * HPC Applications on Clusters * Performance Modeling and Measurement * Novel Algorithms on Clusters * Hybrid programming techniques (MPI+OpenMP, MPI+OpenCL, etc.) * Cluster Benchmarks * Application-level libraries on clusters * Effective use of clusters in novel applications * Performance evaluation tools Area 2: Architecture, Network/Communications, and Management * Energy-efficient cluster architectures * Node and system architecture * Packaging, power and cooling * GPU/ManyCore and heterogeneous clusters * Interconnect/memory architectures * Single system image clusters * Administration and maintenance tools Area 3: Programming and System Software * Cluster System Software/Operating Systems * Cloud-enabling cluster technologies and virtualization * Energy-efficient middleware * Cluster system-level Protocols and APIs * Cluster Security * Resource and job management * Programming and Software Development Environment on Clusters * Fault tolerance and high-availability Area 4: Data, Storage, and Visualization * Cluster Architecture for Big Data storage and processing * Middleware for Big Data management * Cluster-based Cloud Architecture for Big Data * File systems and I/O libraries * Support and Integration of Non-Volatile Memory * Visualization clusters and tiled displays * Big Data visualization tools * Programming models for Big Data processing * Big Data Application studies on cluster architectures *** Submission Guidelines *** Authors are invited to submit papers electronically in PDF format. Submitted manuscripts should be structured as technical papers and may not exceed 10 letter-size (8.5 x 11) pages including figures, tables and references using the IEEE format for conference proceedings. Submissions not conforming to these guidelines may be returned without review. Authors should make sure that their file will print on a printer that uses letter-size (8.5 x 11) paper. The official language of the conference is English. All manuscripts will be reviewed and will be judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference attendees. Paper submissions are limited to 10 pages in 2-column IEEE format including all figures and references. Submitted manuscripts exceeding this limit will be returned without review. For the final camera-ready version, authors with accepted papers may purchase additional pages at the following rates: 200 USD for each of two additional pages. See formatting templates for details: LaTex Package ZIP (http://datasys.cs.iit.edu/events/CCGrid2014/IEEECS_confs_LaTeX.zip) Word Template DOC (http://datasys.cs.iit.edu/events/CCGrid2014/instruct8.5x11x2.doc) and PDF (http://datasys.cs.iit.edu/events/CCGrid2014/instruct8.5x11x2.pdf) Submitted papers must represent original unpublished research that is not currently under review for any other conference or journal. Papers not following these guidelines will be rejected without review and further action may be taken, including (but not limited to) notifications sent to the heads of the institutions of the authors and sponsors of the conference. Submissions received after the due date, exceeding the page limit, or not appropriately structured may not be considered. Authors may contact the conference chairs for more information. The proceedings will be published through the IEEE Computer Society Conference Publishing Services. Please submit your paper via the EasyChair submission system: https://easychair.org/conferences/?conf=ieeecluster2015 *** Journal Special Issue *** The best papers of Cluster 2015 will be included in a Special Issue on advances in topics related to cluster computing of the Elsevier International Journal of Parallel Computing (PARCO), edited by Pavan Balaji, Satoshi Matsuoka, and Michela Taufer. This special issue is dedicated for the papers accepted in the Cluster 2015 conference. The submission to this special issue is by invitation only. *** Important Dates *** ***March 25, 2015*** Papers Submission Deadline (hard deadline) May 9, 2015 Papers Acceptance Notification See other deadlines in the Important Dates page (http://www.mcs.anl.gov/ieeecluster2015/author-information/important-dates) *** Cluster 2015 Program Chair *** Satoshi Matsuoka, Tokyo Institute of Technology (matsu AT is.titech.ac.jp). ---------------------------------------------- ...Follow us on Facebook athttps://www.facebook.com/ieee.cluster ...Follow us on Twitter athttps://twitter.com/IEEECluster ...Follow us on Linkedin at https://www.linkedin.com/groups/IEEE-International-Conference-on-Cluster-7428925 ...Follow us on RenRen athttp://page.renren.com/601871401 ---------------------------------------------- -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= From hategan at mcs.anl.gov Mon Mar 9 15:05:51 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 9 Mar 2015 13:05:51 -0700 Subject: [Swift-devel] array function in standard library Message-ID: <1425931551.20141.7.camel@echo> Hi, I became suspicious that the proposed array functions aren't very well thought of. So I believe that we should not attempt to implement them without further thought. Specifically: 1. T[K] slice(T[K] a, int start, int end) What do start and end signify? If there is some ordering on K, then they could make sense as indices in the ordered set of keys once the array is closed. Unfortunately this requires one to wait for the array to be closed. 2. T[int][K] split(T[K], int n) Same issue. If the splitting is to be deterministic, one needs to wait for the array to be closed. 3. T[int] join(T[K1][K2] a) Same. To some extent, these could be implemented nicely if they worked on "compacted" arrays (i.e. T[int] arrays for which the indices are known to be consecutive). Mihael From tim.g.armstrong at gmail.com Tue Mar 10 09:51:20 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Tue, 10 Mar 2015 07:51:20 -0700 Subject: [Swift-devel] array function in standard library In-Reply-To: <1425931551.20141.7.camel@echo> References: <1425931551.20141.7.camel@echo> Message-ID: I agree. There seems to be a set of array operations that are typically used on dense arrays that depend on knowing where the start of the keyspace is and where the gaps in the keyspace are (knowing whether the array is dense is a special case of that). I can't think of a way to implement them in a way that works for sparse arrays and is intuitive. It would be an option to have part of the contract be that the input array must be dense - we'd just fail if some were missing. That seems like something that almost should be part of the type system rather than a runtime pseudo-type error. In some cases it could degrade more gracefully, e.g just leaving gaps in the output array, but that's not generally viable. I wonder if there are a set of analogous primitives for sparse arrays. e.g. maybe slice(T a[], T begin, T end) just copies any existing keys such that begin <= key <= end to the new array, and leaves any gaps. Anyway, those were some unstructured thoughts - definitely agree that it needs more thought. - Tim On 9 March 2015 at 13:05, Mihael Hategan wrote: > Hi, > > I became suspicious that the proposed array functions aren't very well > thought of. So I believe that we should not attempt to implement them > without further thought. > > Specifically: > > 1. T[K] slice(T[K] a, int start, int end) > > What do start and end signify? If there is some ordering on K, then they > could make sense as indices in the ordered set of keys once the array is > closed. Unfortunately this requires one to wait for the array to be > closed. > > 2. T[int][K] split(T[K], int n) > > Same issue. If the splitting is to be deterministic, one needs to wait > for the array to be closed. > > 3. T[int] join(T[K1][K2] a) > > Same. > > To some extent, these could be implemented nicely if they worked on > "compacted" arrays (i.e. T[int] arrays for which the indices are known > to be consecutive). > > Mihael > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Tue Mar 10 21:59:48 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Tue, 10 Mar 2015 21:59:48 -0500 Subject: [Swift-devel] is restart log required Message-ID: Hi, I was doing some resume tests with trunk on ALCF and noticed the new restart log contains a list of outputs produced by a run. I was wondering if this log is needed at all. In a scenario where user needs to resume a run, the status of the past run could be checked by scanning the output locations specified in the source. This will save the effort of maintaining a file and user will not have to point to the restart log. Just an observation, may be there is more to it than appears at first. --Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Mar 10 22:26:44 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Mar 2015 20:26:44 -0700 Subject: [Swift-devel] is restart log required In-Reply-To: References: Message-ID: <1426044404.3277.1.camel@echo> The presence of a file says nothing about the integrity of that file. It could be a file that was only partially transferred. Mihael On Tue, 2015-03-10 at 21:59 -0500, Ketan Maheshwari wrote: > Hi, > > I was doing some resume tests with trunk on ALCF and noticed the new > restart log contains a list of outputs produced by a run. I was wondering > if this log is needed at all. > > In a scenario where user needs to resume a run, the status of the past run > could be checked by scanning the output locations specified in the source. > This will save the effort of maintaining a file and user will not have to > point to the restart log. > > Just an observation, may be there is more to it than appears at first. > > --Ketan > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From ketan at mcs.anl.gov Wed Mar 11 10:33:05 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Wed, 11 Mar 2015 10:33:05 -0500 Subject: [Swift-devel] second wave of jobs do not start Message-ID: Hi With trunk, coasters on ALCF, I am seeing that after a first wave of jobs finish, the second wave does not start. After the completion of first wave of jobs, the Swift progress text shows jobs in submitted state while the queue (qstat) still shows running status. After a while the queue walltime expires and there are no more new jobs submitted to the queue. Two worker log files are created for the run, possibly the worker shuts down and restarts for a second wave. Attached are the run log and worker logs. Thanks for any help debugging/fixing. -- Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: worker-0311-5002070-000001.log Type: application/octet-stream Size: 1587762 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: worker-0311-5002070-000000.log Type: application/octet-stream Size: 483696 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: run001.tgz Type: application/x-gzip Size: 37688 bytes Desc: not available URL: From ketan at mcs.anl.gov Wed Mar 11 14:16:29 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Wed, 11 Mar 2015 14:16:29 -0500 Subject: [Swift-devel] second wave of jobs do not start In-Reply-To: References: Message-ID: Hi, Please ignore, this was resolved after discussion and debugging with Mike. --Ketan On Wed, Mar 11, 2015 at 10:33 AM, Ketan Maheshwari wrote: > Hi > > With trunk, coasters on ALCF, I am seeing that after a first wave of jobs > finish, the second wave does not start. > > After the completion of first wave of jobs, the Swift progress text shows > jobs in submitted state while the queue (qstat) still shows running status. > After a while the queue walltime expires and there are no more new jobs > submitted to the queue. > > Two worker log files are created for the run, possibly the worker shuts > down and restarts for a second wave. > > Attached are the run log and worker logs. > > Thanks for any help debugging/fixing. > -- > Ketan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Wed Mar 11 14:21:15 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 11 Mar 2015 12:21:15 -0700 Subject: [Swift-devel] second wave of jobs do not start In-Reply-To: References: Message-ID: <1426101675.2433.6.camel@echo> And I'd like to know what the issue was! Mihael On Wed, 2015-03-11 at 14:16 -0500, Ketan Maheshwari wrote: > Hi, > > Please ignore, this was resolved after discussion and debugging with Mike. > > --Ketan > > On Wed, Mar 11, 2015 at 10:33 AM, Ketan Maheshwari > wrote: > > > Hi > > > > With trunk, coasters on ALCF, I am seeing that after a first wave of jobs > > finish, the second wave does not start. > > > > After the completion of first wave of jobs, the Swift progress text shows > > jobs in submitted state while the queue (qstat) still shows running status. > > After a while the queue walltime expires and there are no more new jobs > > submitted to the queue. > > > > Two worker log files are created for the run, possibly the worker shuts > > down and restarts for a second wave. > > > > Attached are the run log and worker logs. > > > > Thanks for any help debugging/fixing. > > -- > > Ketan > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From ketan at mcs.anl.gov Wed Mar 11 14:34:42 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Wed, 11 Mar 2015 14:34:42 -0500 Subject: [Swift-devel] second wave of jobs do not start In-Reply-To: <1426101675.2433.6.camel@echo> References: <1426101675.2433.6.camel@echo> Message-ID: It was related to job and wall times. The maxWalltime was set to 18 minutes and maxJobtime to 20 minutes. After completion of first 8 jobs, coaster thinks there is no more time to accommodate anymore jobs. The run completes after setting maxWalltime=4mins and maxJobtime=60mins. Run was configured to start 2 parallel tasks at a time with 64 total tasks each spanning 2-3 sec. Some notes are below. A few things were not clear from the worker logs which I am trying to study. -- Actually, it was not 1 wave, but 4 waves of 2 tasks were executed (in total 8 tasks). -- The worker starts twice: first instance shuts down after running 2 waves and idling for 3-4 minutes. -- Second instance of worker starts, runs 2 waves of jobs but keeps on idling for more than an half an hour after which I kill the run. In this time, the scheduler job remains in running stage and shuts down when its walltime expires (20 minutes). -- Each worker shows 8 process forked and terminated. Totalling 16 processes in all but we see only 8 tasks. My guess is that for each process, worker forks a watchdog/monitor process (I can be wrong here). -- Ketan On Wed, Mar 11, 2015 at 2:21 PM, Mihael Hategan wrote: > And I'd like to know what the issue was! > > Mihael > > On Wed, 2015-03-11 at 14:16 -0500, Ketan Maheshwari wrote: > > Hi, > > > > Please ignore, this was resolved after discussion and debugging with > Mike. > > > > --Ketan > > > > On Wed, Mar 11, 2015 at 10:33 AM, Ketan Maheshwari > > wrote: > > > > > Hi > > > > > > With trunk, coasters on ALCF, I am seeing that after a first wave of > jobs > > > finish, the second wave does not start. > > > > > > After the completion of first wave of jobs, the Swift progress text > shows > > > jobs in submitted state while the queue (qstat) still shows running > status. > > > After a while the queue walltime expires and there are no more new jobs > > > submitted to the queue. > > > > > > Two worker log files are created for the run, possibly the worker shuts > > > down and restarts for a second wave. > > > > > > Attached are the run log and worker logs. > > > > > > Thanks for any help debugging/fixing. > > > -- > > > Ketan > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsk at uchicago.edu Fri Mar 13 05:00:18 2015 From: dsk at uchicago.edu (Daniel S. Katz) Date: Fri, 13 Mar 2015 06:00:18 -0400 Subject: [Swift-devel] interesting recent papers that cite Swift Message-ID: http://ceur-ws.org/Vol-1330/paper-03.pdf http://link.springer.com/article/10.1007/s10723-015-9329-8 http://www.mcs.anl.gov/~wozniak/papers/Hercules_2014.pdf Dan -- Daniel S. Katz University of Chicago (773) 834-7186 (voice) (773) 834-6818 (fax) d.katz at ieee.org or dsk at uchicago.edu http://www.ci.uchicago.edu/~dsk/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Fri Mar 13 11:38:32 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 13 Mar 2015 11:38:32 -0500 Subject: [Swift-devel] interesting recent papers that cite Swift In-Reply-To: References: Message-ID: Google scholar notifications are handy :). I'm glad there's a new survey paper of scientific workflow systems - the Buyya one from 2005(?) that I had been citing previously was good but out of date. The Hercules work is pretty cool too - that prompted us to add some location features to Swift/T that will be more generally useful - all four combinations of strict/flexible core/node level targeting. - Tim On 13 March 2015 at 05:00, Daniel S. Katz wrote: > http://ceur-ws.org/Vol-1330/paper-03.pdf > > http://link.springer.com/article/10.1007/s10723-015-9329-8 > > http://www.mcs.anl.gov/~wozniak/papers/Hercules_2014.pdf > > Dan > > > -- > Daniel S. Katz > University of Chicago > (773) 834-7186 (voice) > (773) 834-6818 (fax) > d.katz at ieee.org or dsk at uchicago.edu > http://www.ci.uchicago.edu/~dsk/ > > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Fri Mar 13 14:44:24 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 13 Mar 2015 14:44:24 -0500 Subject: [Swift-devel] Polymorphic type notation in Swift/T Message-ID: Here's how I define contains in Swift/T: (boolean o) contains(V A[K], K key) K and V declares that K and V are type variables in the scope of the function definition. I.e. when typechecking call to the function, K and V can be any type, so long as they match in all places. The algorithm for typechecking is to match up input types to the argument types and bind the type variables, e.g. matching a string[int] array to V[K] would bind V=string and K=int, and matching int to K would bind K=int. If there isn't a consistent way to bind K and V, e.g. if one of the K's was a string, then typechecking fails. For split, the function type would look like: T[int][K] split(T[K], int n) - Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilterp at uchicago.edu Sat Mar 14 18:51:44 2015 From: vilterp at uchicago.edu (Pete Vilter) Date: Sat, 14 Mar 2015 23:51:44 +0000 Subject: [Swift-devel] Polymorphic type notation in Swift/T In-Reply-To: References: Message-ID: Hi Tim, This would be very nice to have in Swift/K ? is there a proposal to add it? The syntax is similar to Java static methods; should be familiar to people coming from that background. One question (which applies to Java as well): why is it necessary to declare the type variables at the beginning of the line? Can't the compiler just see what variables are being used in the return and argument types? Thanks, ?Pete On Fri, Mar 13, 2015 at 2:44 PM Tim Armstrong wrote: > Here's how I define contains in Swift/T: > > (boolean o) contains(V A[K], K key) > > K and V declares that K and V are type variables in the scope of the > function definition. I.e. when typechecking call to the function, K and V > can be any type, so long as they match in all places. > > The algorithm for typechecking is to match up input types to the argument > types and bind the type variables, e.g. matching a string[int] array to > V[K] would bind V=string and K=int, and matching int to K would bind > K=int. If there isn't a consistent way to bind K and V, e.g. if one of the > K's was a string, then typechecking fails. > > > For split, the function type would look like: > > T[int][K] split(T[K], int n) > > - Tim > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Sun Mar 15 13:16:13 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Sun, 15 Mar 2015 13:16:13 -0500 Subject: [Swift-devel] Polymorphic type notation in Swift/T In-Reply-To: References: Message-ID: This is only currently supported for external Tcl functions, btw, not regular Swift composite functions. It's possible to do what you suggest - assume that any unknown types in the type signature are type variables. It would be pretty straightforward to implement too. The downside is that can lead to surprising behaviour from the user's perspective. For example, suppose I declare a function with this type: f(Int x) Int isn't the name of a type (the real integer type is int), so it would be inferred to be a type variable, and then you have a function that looks like it accepts ints, but actually accepts any input argument. There are other scenarios too - e.g. if a type is defined in an imported module but you forget to import the module, suddenly you have a type variable where you thought you had a regular type. I guess the way to think about it is that you're asking the programmer to add a little redundant information about their intent that then allows the compiler to check that their code is doing what they intended. - Tim On 14 March 2015 at 18:51, Pete Vilter wrote: > Hi Tim, > This would be very nice to have in Swift/K ? is there a proposal to add > it? The syntax is similar to Java static methods; should be familiar to > people coming from that background. One question (which applies to Java as > well): why is it necessary to declare the type variables at the beginning > of the line? Can't the compiler just see what variables are being used in > the return and argument types? > Thanks, > ?Pete > > On Fri, Mar 13, 2015 at 2:44 PM Tim Armstrong > wrote: > >> Here's how I define contains in Swift/T: >> >> (boolean o) contains(V A[K], K key) >> >> K and V declares that K and V are type variables in the scope of the >> function definition. I.e. when typechecking call to the function, K and V >> can be any type, so long as they match in all places. >> >> The algorithm for typechecking is to match up input types to the argument >> types and bind the type variables, e.g. matching a string[int] array to >> V[K] would bind V=string and K=int, and matching int to K would bind >> K=int. If there isn't a consistent way to bind K and V, e.g. if one of the >> K's was a string, then typechecking fails. >> >> >> For split, the function type would look like: >> >> T[int][K] split(T[K], int n) >> >> - Tim >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilterp at uchicago.edu Sun Mar 15 14:01:07 2015 From: vilterp at uchicago.edu (Pete Vilter) Date: Sun, 15 Mar 2015 19:01:07 +0000 Subject: [Swift-devel] Polymorphic type notation in Swift/T In-Reply-To: References: Message-ID: Thanks Tim, that reasoning makes sense. On Sun, Mar 15, 2015 at 1:16 PM Tim Armstrong wrote: > This is only currently supported for external Tcl functions, btw, not > regular Swift composite functions. > > It's possible to do what you suggest - assume that any unknown types in > the type signature are type variables. It would be pretty straightforward > to implement too. The downside is that can lead to surprising behaviour > from the user's perspective. > > For example, suppose I declare a function with this type: > > f(Int x) > > Int isn't the name of a type (the real integer type is int), so it would > be inferred to be a type variable, and then you have a function that looks > like it accepts ints, but actually accepts any input argument. > > There are other scenarios too - e.g. if a type is defined in an imported > module but you forget to import the module, suddenly you have a type > variable where you thought you had a regular type. > > I guess the way to think about it is that you're asking the programmer to > add a little redundant information about their intent that then allows the > compiler to check that their code is doing what they intended. > > - Tim > > On 14 March 2015 at 18:51, Pete Vilter wrote: > >> Hi Tim, >> This would be very nice to have in Swift/K ? is there a proposal to add >> it? The syntax is similar to Java static methods; should be familiar to >> people coming from that background. One question (which applies to Java as >> well): why is it necessary to declare the type variables at the beginning >> of the line? Can't the compiler just see what variables are being used in >> the return and argument types? >> Thanks, >> ?Pete >> >> On Fri, Mar 13, 2015 at 2:44 PM Tim Armstrong >> wrote: >> >>> Here's how I define contains in Swift/T: >>> >>> (boolean o) contains(V A[K], K key) >>> >>> K and V declares that K and V are type variables in the scope of the >>> function definition. I.e. when typechecking call to the function, K and V >>> can be any type, so long as they match in all places. >>> >>> The algorithm for typechecking is to match up input types to the >>> argument types and bind the type variables, e.g. matching a string[int] >>> array to V[K] would bind V=string and K=int, and matching int to K would >>> bind K=int. If there isn't a consistent way to bind K and V, e.g. if one >>> of the K's was a string, then typechecking fails. >>> >>> >>> For split, the function type would look like: >>> >>> T[int][K] split(T[K], int n) >>> >>> - Tim >>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Fri Mar 20 10:54:08 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 20 Mar 2015 10:54:08 -0500 Subject: [Swift-devel] Task matching queue scalability benchmark Message-ID: Hi All, I just wanted to share some benchmark results I did that may be of interest to the team. One thing Justin and I have been working on for quite a while is improving the core ADLB data structures that match up tasks to workers. The efficiency of these affects task throughput in Swift/T a lot, and the scalability as a lot of tasks get enqueued can make a big difference on many workloads. It's actually pretty complex getting the data structures right, since we need to support task priorities plus task types plus various kinds of locality, plus matching in both directions (incoming tasks to idle workers, incoming worker requests to tasks), plus we want very high efficiency - a few hundred thousand tasks per second per server means a few microseconds per match. Justin did the initlal rewrite in 2012 (I think) to replace the linked lists originally used (which don't scale well). We've gone back and forth since then making further improvements. Recently I made some further changes for locality-aware matching that resulted in the current performance results in the benchmark. The short version is: everything performs great (100s of nanoseconds per task) even with 1 million tasks in the queue and using all locality-aware matching features. I did experiments to quantify the results of all that work. The experiment simulates incoming tasks and requests for tasks and times how long the operations on the data structures take (we don't include MPI and other overheads, which add 100%+ overhead in practice). The experiment results are divided up in a few ways: EQUAL/UNIFORM_RANDOM - whether priorities are all equal or are randomly chosen UNTARGETED/TARGETED - whether it can go to any worker, or a specified subset of workers RANK/NODE - whether a rank or all ranks on a node are targeted SOFT - if the task can go to a different rank if the targeted one is busy. On the x axis we have the queue size - how many tasks there are in the queue. On the y axis is the average time per operation in nanoseconds. It requires two operations to match a task, so double that time to match a task Overall everything is very efficient - worst case is 0.3 of a microsecond, most are 50 to 100 nanoseconds. Equal priorities show performance independent of queue size, while random priorities scales logarithmically and perform well even with a million tasks in the queue. There's some constant overhead associated with node targeting and soft targeting since the logic is slightly more complicated. - Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: q-scalability-beagle.png Type: image/png Size: 78742 bytes Desc: not available URL: From ketan at mcs.anl.gov Fri Mar 20 23:33:51 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Fri, 20 Mar 2015 23:33:51 -0500 Subject: [Swift-devel] swift write once array/struct Message-ID: Hi, So, in the meeting today, I was curious about wether the Swift write once array/struct property is enforced via some external function or is it intrinsic to the array/struct. In other words, is the array/struct implemented in such a way that it will automatically not allow anything to write once its i-th item has been written or some external entity keeps track of this property and gets triggered every time a write happens. I thought with intrinsic property, write operations will be atomic and free of deadlocks as opposed to externally tracked/enforced property. -- Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Sat Mar 21 09:12:04 2015 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sat, 21 Mar 2015 09:12:04 -0500 Subject: [Swift-devel] IEEE Cluster 2015 hard deadline approaching - March 25, 2015 (AOE) Message-ID: <550D7C34.1060405@cs.iit.edu> March 20, 2015 Release IEEE International Conference on Cluster Computing September 8-11, 2015 Chicago, IL, USA http://www.mcs.anl.gov/ieeecluster2015/ *** NEW!!!! Important Dates *** March 25, 2015 (AOE) Papers Submission Deadline May 9, 2015 Papers Acceptance Notification See other deadlines in the Important Dates page (http://www.mcs.anl.gov/ieeecluster2015/author-information/important-dates) CLUSTER 2015 CALL FOR PAPERS Following the successes of the series of Cluster conferences, for 2015 we solicit high-quality original papers presenting work that advances the state-of-the-art in clusters and closely related fields. All papers will be rigorously peer-reviewed for their originality, technical depth and correctness, potential impact, relevance to the conference, and quality of presentation. Research papers must clearly demonstrate research contributions and novelty, while papers reporting experience must clearly describe lessons learned and impact, along with the utility of the approach compared to the ones in the past. *** Paper Tracks *** Area 1: Application, Algorithms, and Libraries * HPC Applications on Clusters * Performance Modeling and Measurement * Novel Algorithms on Clusters * Hybrid programming techniques (MPI+OpenMP, MPI+OpenCL, etc.) * Cluster Benchmarks * Application-level libraries on clusters * Effective use of clusters in novel applications * Performance evaluation tools Area 2: Architecture, Network/Communications, and Management * Energy-efficient cluster architectures * Node and system architecture * Packaging, power and cooling * GPU/ManyCore and heterogeneous clusters * Interconnect/memory architectures * Single system image clusters * Administration and maintenance tools Area 3: Programming and System Software * Cluster System Software/Operating Systems * Cloud-enabling cluster technologies and virtualization * Energy-efficient middleware * Cluster system-level Protocols and APIs * Cluster Security * Resource and job management * Programming and Software Development Environment on Clusters * Fault tolerance and high-availability Area 4: Data, Storage, and Visualization * Cluster Architecture for Big Data storage and processing * Middleware for Big Data management * Cluster-based Cloud Architecture for Big Data * File systems and I/O libraries * Support and Integration of Non-Volatile Memory * Visualization clusters and tiled displays * Big Data visualization tools * Programming models for Big Data processing * Big Data Application studies on cluster architectures *** Submission Guidelines *** Authors are invited to submit papers electronically in PDF format. Submitted manuscripts should be structured as technical papers and may not exceed 10 letter-size (8.5 x 11) pages including figures, tables and references using the IEEE format for conference proceedings. Submissions not conforming to these guidelines may be returned without review. Authors should make sure that their file will print on a printer that uses letter-size (8.5 x 11) paper. The official language of the conference is English. All manuscripts will be reviewed and will be judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference attendees. Paper submissions are limited to 10 pages in 2-column IEEE format including all figures and references. Submitted manuscripts exceeding this limit will be returned without review. For the final camera-ready version, authors with accepted papers may purchase additional pages at the following rates: 200 USD for each of two additional pages. See formatting templates for details: LaTex Package ZIP (http://datasys.cs.iit.edu/events/CCGrid2014/IEEECS_confs_LaTeX.zip) Word Template DOC (http://datasys.cs.iit.edu/events/CCGrid2014/instruct8.5x11x2.doc) and PDF (http://datasys.cs.iit.edu/events/CCGrid2014/instruct8.5x11x2.pdf) Submitted papers must represent original unpublished research that is not currently under review for any other conference or journal. Papers not following these guidelines will be rejected without review and further action may be taken, including (but not limited to) notifications sent to the heads of the institutions of the authors and sponsors of the conference. Submissions received after the due date, exceeding the page limit, or not appropriately structured may not be considered. Authors may contact the conference chairs for more information. The proceedings will be published through the IEEE Computer Society Conference Publishing Services. Please submit your paper via the EasyChair submission system: https://easychair.org/conferences/?conf=ieeecluster2015 *** Journal Special Issue *** The best papers of Cluster 2015 will be included in a Special Issue on advances in topics related to cluster computing of the Elsevier International Journal of Parallel Computing (PARCO), edited by Pavan Balaji, Satoshi Matsuoka, and Michela Taufer. This special issue is dedicated for the papers accepted in the Cluster 2015 conference. The submission to this special issue is by invitation only. *** Cluster 2015 Program Chair *** Satoshi Matsuoka, Tokyo Institute of Technology (matsu AT is.titech.ac.jp). ---------------------------------------------- ...Follow us on Facebook athttps://www.facebook.com/ieee.cluster ...Follow us on Twitter athttps://twitter.com/IEEECluster ...Follow us on Linkedin at https://www.linkedin.com/groups/IEEE-International-Conference-on-Cluster-7428925 ...Follow us on RenRen athttp://page.renren.com/601871401 ---------------------------------------------- -- -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Sat Mar 21 10:06:42 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Sat, 21 Mar 2015 10:06:42 -0500 Subject: [Swift-devel] swift write once array/struct In-Reply-To: References: Message-ID: Hi Ketan, I'm not sure that I fully understand the distinction between internal/external implementation. In terms of externally observable behaviour, writing an already written cell will cause a runtime error immediately. There's no way you can cause a deadlock or observe inconsistent values. Deadlocks with array cells should only happen if something is waiting for a cell that has been assigned 0 times. If you're curious, i copied and pasted the relevant code fragment for Swift/T (it's in data.c in lb). It's relatively straightforward, even though it's handling a few cases. Essentially it's if (A[i] exists) return error code;. The data structure is only accessed by a single thread so no race conditions are possible. I believe there's something similar in swift/k. - Tim // Does the link already exist? adlb_container_val t = NULL; bool found = container_lookup(c, curr_sub, &t); if (found && (value == NULL || t != NULL)) { // Can overwrite reserved (unlinked) entries with actual data, but // cannot double reserve entries. // Don't print error by default: caller may want to handle DEBUG("already exists: "ADLB_PRIDSUB, ADLB_PRIDSUB_ARGS(id, d->symbol, subscript)); return ADLB_DATA_ERROR_DOUBLE_WRITE; } // Following code actually assigns the array cell ... On 20 March 2015 at 23:33, Ketan Maheshwari wrote: > Hi, > > So, in the meeting today, I was curious about wether the Swift write once > array/struct property is enforced via some external function or is it > intrinsic to the array/struct. > > In other words, is the array/struct implemented in such a way that it > will automatically not allow anything to write once its i-th item has been > written or some external entity keeps track of this property and gets > triggered every time a write happens. > > I thought with intrinsic property, write operations will be atomic and > free of deadlocks as opposed to externally tracked/enforced property. > > -- > Ketan > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Sat Mar 21 10:14:16 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Sat, 21 Mar 2015 10:14:16 -0500 Subject: [Swift-devel] swift write once array/struct In-Reply-To: References: Message-ID: Also, if you're curious about semantics of Swift data structures I've been doing some work on that recently, would be happy to talk about it in more detail. - Tim On 21 March 2015 at 10:06, Tim Armstrong wrote: > Hi Ketan, > I'm not sure that I fully understand the distinction between > internal/external implementation. In terms of externally observable > behaviour, writing an already written cell will cause a runtime error > immediately. There's no way you can cause a deadlock or observe > inconsistent values. Deadlocks with array cells should only happen if > something is waiting for a cell that has been assigned 0 times. > > If you're curious, i copied and pasted the relevant code fragment for > Swift/T (it's in data.c in lb). It's relatively straightforward, even > though it's handling a few cases. Essentially it's if (A[i] exists) > return error code;. The data structure is only accessed by a single > thread so no race conditions are possible. > > I believe there's something similar in swift/k. > > - Tim > > // Does the link already exist? > adlb_container_val t = NULL; > bool found = container_lookup(c, curr_sub, &t); > > if (found && (value == NULL || t != NULL)) > { > // Can overwrite reserved (unlinked) entries with actual data, but > // cannot double reserve entries. > > // Don't print error by default: caller may want to handle > DEBUG("already exists: "ADLB_PRIDSUB, > ADLB_PRIDSUB_ARGS(id, d->symbol, subscript)); > return ADLB_DATA_ERROR_DOUBLE_WRITE; > } > // Following code actually assigns the array cell > ... > > > > > > On 20 March 2015 at 23:33, Ketan Maheshwari wrote: > >> Hi, >> >> So, in the meeting today, I was curious about wether the Swift write once >> array/struct property is enforced via some external function or is it >> intrinsic to the array/struct. >> >> In other words, is the array/struct implemented in such a way that it >> will automatically not allow anything to write once its i-th item has been >> written or some external entity keeps track of this property and gets >> triggered every time a write happens. >> >> I thought with intrinsic property, write operations will be atomic and >> free of deadlocks as opposed to externally tracked/enforced property. >> >> -- >> Ketan >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Sat Mar 21 10:38:58 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Sat, 21 Mar 2015 10:38:58 -0500 Subject: [Swift-devel] swift write once array/struct In-Reply-To: References: Message-ID: Hi Tim, Thanks for the clarification and code snippet. To clarify, in my understanding, the implementation is 'extrinsic'. To be intrinsic, I imagined the array is implemented as an object which has a function that monitors and controls its write-once property and gets invoked every time anything is written to any element of the array. Very rudimentary code below to clarify the idea: bool checklock(val_t val){ if (self.item) return ERRNOWRITE; dowrite(self.item, val); return OK; } About Swift semantics work: thanks, I will try to take a look and comment if I can. -- Ketan On Sat, Mar 21, 2015 at 10:14 AM, Tim Armstrong wrote: > Also, if you're curious about semantics of Swift data structures I've been > doing some work on that recently, would be happy to talk about it in more > detail. > > - Tim > > On 21 March 2015 at 10:06, Tim Armstrong > wrote: > >> Hi Ketan, >> I'm not sure that I fully understand the distinction between >> internal/external implementation. In terms of externally observable >> behaviour, writing an already written cell will cause a runtime error >> immediately. There's no way you can cause a deadlock or observe >> inconsistent values. Deadlocks with array cells should only happen if >> something is waiting for a cell that has been assigned 0 times. >> >> If you're curious, i copied and pasted the relevant code fragment for >> Swift/T (it's in data.c in lb). It's relatively straightforward, even >> though it's handling a few cases. Essentially it's if (A[i] exists) >> return error code;. The data structure is only accessed by a single >> thread so no race conditions are possible. >> >> I believe there's something similar in swift/k. >> >> - Tim >> >> // Does the link already exist? >> adlb_container_val t = NULL; >> bool found = container_lookup(c, curr_sub, &t); >> >> if (found && (value == NULL || t != NULL)) >> { >> // Can overwrite reserved (unlinked) entries with actual data, but >> // cannot double reserve entries. >> >> // Don't print error by default: caller may want to handle >> DEBUG("already exists: "ADLB_PRIDSUB, >> ADLB_PRIDSUB_ARGS(id, d->symbol, subscript)); >> return ADLB_DATA_ERROR_DOUBLE_WRITE; >> } >> // Following code actually assigns the array cell >> ... >> >> >> >> >> >> On 20 March 2015 at 23:33, Ketan Maheshwari wrote: >> >>> Hi, >>> >>> So, in the meeting today, I was curious about wether the Swift write >>> once array/struct property is enforced via some external function or is it >>> intrinsic to the array/struct. >>> >>> In other words, is the array/struct implemented in such a way that it >>> will automatically not allow anything to write once its i-th item has been >>> written or some external entity keeps track of this property and gets >>> triggered every time a write happens. >>> >>> I thought with intrinsic property, write operations will be atomic and >>> free of deadlocks as opposed to externally tracked/enforced property. >>> >>> -- >>> Ketan >>> >>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>> >>> >> > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Sat Mar 21 11:30:48 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Sat, 21 Mar 2015 11:30:48 -0500 Subject: [Swift-devel] swift write once array/struct In-Reply-To: References: Message-ID: I still don't fully understand what distinction you're drawing. Your example seems functionally identical to the Swift/T example - on all codepaths where an array cell might be written, the code checks whether or not it was previously assigned before the second assignment happens. >From the point of view of language semantics, the only relevant question is "what happens when an array cell is assigned twice". 1. We can't always/don't detect it at compile time, so the program still runs. 2. The first assignment A[i] = x; completes fine 3. Any reads before the second assignment return x; 4. The second assignment A[i] = y; causes a runtime error and stops the program immediately It's possible that the assignments happen in the opposite order, so x and y can be swapped. I'm not sure how the implementation details, e.g. how it's divided into objects, matters so long as it implements the above semantics correctly. Is the question whether the second assignment atomically causes a runtime error? It does in the current implementation. - Tim On 21 March 2015 at 10:38, Ketan Maheshwari wrote: > Hi Tim, > > Thanks for the clarification and code snippet. To clarify, in my > understanding, the implementation is 'extrinsic'. > > To be intrinsic, I imagined the array is implemented as an object which > has a function that monitors and controls its write-once property and gets > invoked every time anything is written to any element of the array. > > Very rudimentary code below to clarify the idea: > > bool checklock(val_t val){ > if (self.item) return ERRNOWRITE; > dowrite(self.item, val); > return OK; > } > > About Swift semantics work: thanks, I will try to take a look and comment > if I can. > > -- > Ketan > > On Sat, Mar 21, 2015 at 10:14 AM, Tim Armstrong > wrote: > >> Also, if you're curious about semantics of Swift data structures I've >> been doing some work on that recently, would be happy to talk about it in >> more detail. >> >> - Tim >> >> On 21 March 2015 at 10:06, Tim Armstrong >> wrote: >> >>> Hi Ketan, >>> I'm not sure that I fully understand the distinction between >>> internal/external implementation. In terms of externally observable >>> behaviour, writing an already written cell will cause a runtime error >>> immediately. There's no way you can cause a deadlock or observe >>> inconsistent values. Deadlocks with array cells should only happen if >>> something is waiting for a cell that has been assigned 0 times. >>> >>> If you're curious, i copied and pasted the relevant code fragment for >>> Swift/T (it's in data.c in lb). It's relatively straightforward, even >>> though it's handling a few cases. Essentially it's if (A[i] exists) >>> return error code;. The data structure is only accessed by a single >>> thread so no race conditions are possible. >>> >>> I believe there's something similar in swift/k. >>> >>> - Tim >>> >>> // Does the link already exist? >>> adlb_container_val t = NULL; >>> bool found = container_lookup(c, curr_sub, &t); >>> >>> if (found && (value == NULL || t != NULL)) >>> { >>> // Can overwrite reserved (unlinked) entries with actual data, >>> but >>> // cannot double reserve entries. >>> >>> // Don't print error by default: caller may want to handle >>> DEBUG("already exists: "ADLB_PRIDSUB, >>> ADLB_PRIDSUB_ARGS(id, d->symbol, subscript)); >>> return ADLB_DATA_ERROR_DOUBLE_WRITE; >>> } >>> // Following code actually assigns the array cell >>> ... >>> >>> >>> >>> >>> >>> On 20 March 2015 at 23:33, Ketan Maheshwari wrote: >>> >>>> Hi, >>>> >>>> So, in the meeting today, I was curious about wether the Swift write >>>> once array/struct property is enforced via some external function or is it >>>> intrinsic to the array/struct. >>>> >>>> In other words, is the array/struct implemented in such a way that it >>>> will automatically not allow anything to write once its i-th item has been >>>> written or some external entity keeps track of this property and gets >>>> triggered every time a write happens. >>>> >>>> I thought with intrinsic property, write operations will be atomic and >>>> free of deadlocks as opposed to externally tracked/enforced property. >>>> >>>> -- >>>> Ketan >>>> >>>> _______________________________________________ >>>> Swift-devel mailing list >>>> Swift-devel at ci.uchicago.edu >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>> >>>> >>> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Sat Mar 21 11:40:06 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Sat, 21 Mar 2015 11:40:06 -0500 Subject: [Swift-devel] swift write once array/struct In-Reply-To: References: Message-ID: It just occurred to me - are you thinking about it in terms of some actor-based or object-based model of concurrency? - Tim On 21 March 2015 at 11:30, Tim Armstrong wrote: > I still don't fully understand what distinction you're drawing. Your > example seems functionally identical to the Swift/T example - on all > codepaths where an array cell might be written, the code checks whether or > not it was previously assigned before the second assignment happens. > > From the point of view of language semantics, the only relevant question > is "what happens when an array cell is assigned twice". > > 1. We can't always/don't detect it at compile time, so the program still > runs. > 2. The first assignment A[i] = x; completes fine > 3. Any reads before the second assignment return x; > 4. The second assignment A[i] = y; causes a runtime error and stops the > program immediately > > It's possible that the assignments happen in the opposite order, so x and > y can be swapped. > > I'm not sure how the implementation details, e.g. how it's divided into > objects, matters so long as it implements the above semantics correctly. > Is the question whether the second assignment atomically causes a runtime > error? It does in the current implementation. > > - Tim > > > > On 21 March 2015 at 10:38, Ketan Maheshwari wrote: > >> Hi Tim, >> >> Thanks for the clarification and code snippet. To clarify, in my >> understanding, the implementation is 'extrinsic'. >> >> To be intrinsic, I imagined the array is implemented as an object which >> has a function that monitors and controls its write-once property and gets >> invoked every time anything is written to any element of the array. >> >> Very rudimentary code below to clarify the idea: >> >> bool checklock(val_t val){ >> if (self.item) return ERRNOWRITE; >> dowrite(self.item, val); >> return OK; >> } >> >> About Swift semantics work: thanks, I will try to take a look and comment >> if I can. >> >> -- >> Ketan >> >> On Sat, Mar 21, 2015 at 10:14 AM, Tim Armstrong < >> tim.g.armstrong at gmail.com> wrote: >> >>> Also, if you're curious about semantics of Swift data structures I've >>> been doing some work on that recently, would be happy to talk about it in >>> more detail. >>> >>> - Tim >>> >>> On 21 March 2015 at 10:06, Tim Armstrong >>> wrote: >>> >>>> Hi Ketan, >>>> I'm not sure that I fully understand the distinction between >>>> internal/external implementation. In terms of externally observable >>>> behaviour, writing an already written cell will cause a runtime error >>>> immediately. There's no way you can cause a deadlock or observe >>>> inconsistent values. Deadlocks with array cells should only happen if >>>> something is waiting for a cell that has been assigned 0 times. >>>> >>>> If you're curious, i copied and pasted the relevant code fragment for >>>> Swift/T (it's in data.c in lb). It's relatively straightforward, even >>>> though it's handling a few cases. Essentially it's if (A[i] exists) >>>> return error code;. The data structure is only accessed by a single >>>> thread so no race conditions are possible. >>>> >>>> I believe there's something similar in swift/k. >>>> >>>> - Tim >>>> >>>> // Does the link already exist? >>>> adlb_container_val t = NULL; >>>> bool found = container_lookup(c, curr_sub, &t); >>>> >>>> if (found && (value == NULL || t != NULL)) >>>> { >>>> // Can overwrite reserved (unlinked) entries with actual data, >>>> but >>>> // cannot double reserve entries. >>>> >>>> // Don't print error by default: caller may want to handle >>>> DEBUG("already exists: "ADLB_PRIDSUB, >>>> ADLB_PRIDSUB_ARGS(id, d->symbol, subscript)); >>>> return ADLB_DATA_ERROR_DOUBLE_WRITE; >>>> } >>>> // Following code actually assigns the array cell >>>> ... >>>> >>>> >>>> >>>> >>>> >>>> On 20 March 2015 at 23:33, Ketan Maheshwari wrote: >>>> >>>>> Hi, >>>>> >>>>> So, in the meeting today, I was curious about wether the Swift write >>>>> once array/struct property is enforced via some external function or is it >>>>> intrinsic to the array/struct. >>>>> >>>>> In other words, is the array/struct implemented in such a way that it >>>>> will automatically not allow anything to write once its i-th item has been >>>>> written or some external entity keeps track of this property and gets >>>>> triggered every time a write happens. >>>>> >>>>> I thought with intrinsic property, write operations will be atomic and >>>>> free of deadlocks as opposed to externally tracked/enforced property. >>>>> >>>>> -- >>>>> Ketan >>>>> >>>>> _______________________________________________ >>>>> Swift-devel mailing list >>>>> Swift-devel at ci.uchicago.edu >>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>>> >>>>> >>>> >>> >>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sat Mar 21 12:28:12 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 21 Mar 2015 10:28:12 -0700 Subject: [Swift-devel] swift write once array/struct In-Reply-To: References: Message-ID: <1426958892.19708.9.camel@echo> On Sat, 2015-03-21 at 10:38 -0500, Ketan Maheshwari wrote: > Hi Tim, > > Thanks for the clarification and code snippet. To clarify, in my > understanding, the implementation is 'extrinsic'. > > To be intrinsic, I imagined the array is implemented as an object which has > a function that monitors and controls its write-once property and gets > invoked every time anything is written to any element of the array. The array is implemented as you describe above. I'm not really sure what extrinsic would be. Calls to an object's methods aren't automatically synchronized or atomic, so I don't think that assumption in your earlier email is correct. Even if they were, having this particular method atomic does not guarantee that there would be no deadlocks. A user can write a[0] = f(a[1]); a[1] = f(a[0]); Mihael From ketan at mcs.anl.gov Sat Mar 21 14:16:16 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Sat, 21 Mar 2015 14:16:16 -0500 Subject: [Swift-devel] swift write once array/struct In-Reply-To: <4eaff3ab965d4f36aa130b5b50c58121@NAGURSKI.anl.gov> References: <4eaff3ab965d4f36aa130b5b50c58121@NAGURSKI.anl.gov> Message-ID: Thanks Mihael, that clarifies it. Tim, I was thinking from the point of view of avoiding deadlocks. I understand the semantics would remain same. I was thinking a self contained array/struct in terms of state management (may be it is called actor based impl, not sure) would be more robust in terms of avoiding deadlocks. -- Ketan On Sat, Mar 21, 2015 at 12:28 PM, Hategan-Marandiuc, Philip M. < hategan at mcs.anl.gov> wrote: > On Sat, 2015-03-21 at 10:38 -0500, Ketan Maheshwari wrote: > > Hi Tim, > > > > Thanks for the clarification and code snippet. To clarify, in my > > understanding, the implementation is 'extrinsic'. > > > > To be intrinsic, I imagined the array is implemented as an object which > has > > a function that monitors and controls its write-once property and gets > > invoked every time anything is written to any element of the array. > > The array is implemented as you describe above. I'm not really sure what > extrinsic would be. > > Calls to an object's methods aren't automatically synchronized or > atomic, so I don't think that assumption in your earlier email is > correct. > > Even if they were, having this particular method atomic does not > guarantee that there would be no deadlocks. A user can write a[0] = > f(a[1]); a[1] = f(a[0]); > > Mihael > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sat Mar 21 14:34:13 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 21 Mar 2015 12:34:13 -0700 Subject: [Swift-devel] swift write once array/struct In-Reply-To: References: <4eaff3ab965d4f36aa130b5b50c58121@NAGURSKI.anl.gov> Message-ID: <1426966453.21072.4.camel@echo> One more comment. This is not a deadlock that comes from a faulty implementation, so better software engineering won't fix it. This is an algorithmic issue in that the language allows more flexibility that the algorithms implementing it can analyze to determine when something that is theoretically deadlock-free would in practice be deadlock-free. So there are corner cases where the implementation fails not because of something that was missed, but because I don't quite know how to do it better. Mihael On Sat, 2015-03-21 at 14:16 -0500, Ketan Maheshwari wrote: > Thanks Mihael, that clarifies it. > > Tim, I was thinking from the point of view of avoiding deadlocks. I > understand the semantics would remain same. I was thinking a self contained > array/struct in terms of state management (may be it is called actor based > impl, not sure) would be more robust in terms of avoiding deadlocks. > > -- > Ketan > > On Sat, Mar 21, 2015 at 12:28 PM, Hategan-Marandiuc, Philip M. < > hategan at mcs.anl.gov> wrote: > > > On Sat, 2015-03-21 at 10:38 -0500, Ketan Maheshwari wrote: > > > Hi Tim, > > > > > > Thanks for the clarification and code snippet. To clarify, in my > > > understanding, the implementation is 'extrinsic'. > > > > > > To be intrinsic, I imagined the array is implemented as an object which > > has > > > a function that monitors and controls its write-once property and gets > > > invoked every time anything is written to any element of the array. > > > > The array is implemented as you describe above. I'm not really sure what > > extrinsic would be. > > > > Calls to an object's methods aren't automatically synchronized or > > atomic, so I don't think that assumption in your earlier email is > > correct. > > > > Even if they were, having this particular method atomic does not > > guarantee that there would be no deadlocks. A user can write a[0] = > > f(a[1]); a[1] = f(a[0]); > > > > Mihael > > > > From tim.g.armstrong at gmail.com Sat Mar 21 15:26:03 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Sat, 21 Mar 2015 15:26:03 -0500 Subject: [Swift-devel] swift write once array/struct In-Reply-To: <1426966453.21072.4.camel@echo> References: <4eaff3ab965d4f36aa130b5b50c58121@NAGURSKI.anl.gov> <1426966453.21072.4.camel@echo> Message-ID: Agreed, with some of these scripts there are a few questions: * In what scenarios are there a consistent set of values that could be assigned to data structures? * In what scenarios does the language specification guarantee that the program makes progress and those values are actually assigned? * In what scenarios does the implementation actually assign the values? The last two are kind of blurred in Swift since it's not precisely specified. E.g. this script is not sensible because there's no consistent value for the size of the array. int A[]; if (size(A) == 0) { A[0] = 1; } This script could reasonably have size(A) == 1 but I don't think it's reasonable to guarantee progress. int A[]; if (size(A) == 1) { A[0] = 1; } This script could reasonably have size(A) == 1, and it might be reasonable to guarantee progress, since you could have some system where "intent" to assign the subscript is registered before the subscript is assigned. I.e. it goes from EMPTY -> WILL_BE_FULL -> FULL. int A[]; A[0] = size(A); Another example might be int A[]; if (contains(A, 0)) { A[1] = x; } This example could make progress if the language was smart enough to infer that A[0] wasn't assigned in the conditional. I think to better pin down the specification/implementation we probably need some kind of effect system that models more precisely which bits of code have what effect on the data structures - http://en.wikipedia.org/wiki/Effect_system We have a prototypical effect system with the write reference counting but you can have more granular ones where you track which parts of the array might be modified, etc. - Tim On 21 March 2015 at 14:34, Mihael Hategan wrote: > One more comment. > > This is not a deadlock that comes from a faulty implementation, so > better software engineering won't fix it. > > This is an algorithmic issue in that the language allows more > flexibility that the algorithms implementing it can analyze to determine > when something that is theoretically deadlock-free would in practice be > deadlock-free. So there are corner cases where the implementation fails > not because of something that was missed, but because I don't quite know > how to do it better. > > Mihael > > On Sat, 2015-03-21 at 14:16 -0500, Ketan Maheshwari wrote: > > Thanks Mihael, that clarifies it. > > > > Tim, I was thinking from the point of view of avoiding deadlocks. I > > understand the semantics would remain same. I was thinking a self > contained > > array/struct in terms of state management (may be it is called actor > based > > impl, not sure) would be more robust in terms of avoiding deadlocks. > > > > -- > > Ketan > > > > On Sat, Mar 21, 2015 at 12:28 PM, Hategan-Marandiuc, Philip M. < > > hategan at mcs.anl.gov> wrote: > > > > > On Sat, 2015-03-21 at 10:38 -0500, Ketan Maheshwari wrote: > > > > Hi Tim, > > > > > > > > Thanks for the clarification and code snippet. To clarify, in my > > > > understanding, the implementation is 'extrinsic'. > > > > > > > > To be intrinsic, I imagined the array is implemented as an object > which > > > has > > > > a function that monitors and controls its write-once property and > gets > > > > invoked every time anything is written to any element of the array. > > > > > > The array is implemented as you describe above. I'm not really sure > what > > > extrinsic would be. > > > > > > Calls to an object's methods aren't automatically synchronized or > > > atomic, so I don't think that assumption in your earlier email is > > > correct. > > > > > > Even if they were, having this particular method atomic does not > > > guarantee that there would be no deadlocks. A user can write a[0] = > > > f(a[1]); a[1] = f(a[0]); > > > > > > Mihael > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sun Mar 22 13:50:03 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 22 Mar 2015 11:50:03 -0700 Subject: [Swift-devel] swift write once array/struct In-Reply-To: References: <4eaff3ab965d4f36aa130b5b50c58121@NAGURSKI.anl.gov> <1426966453.21072.4.camel@echo> Message-ID: <1427050203.30613.8.camel@echo> On Sat, 2015-03-21 at 15:26 -0500, Tim Armstrong wrote: > E.g. this script is not sensible because there's no consistent value for > the size of the array. > > int A[]; > if (size(A) == 0) { > A[0] = 1; > } Right. Which is why size(A) is only defined on a closed array in K. So the above example is a deadlock that could even possibly be detected at compile time. > > This script could reasonably have size(A) == 1 but I don't think it's > reasonable to guarantee progress. > int A[]; > if (size(A) == 1) { > A[0] = 1; > } > > This script could reasonably have size(A) == 1, and it might be reasonable > to guarantee progress, since you could have some system where "intent" to > assign the subscript is registered before the subscript is assigned. I.e. > it goes from EMPTY -> WILL_BE_FULL -> FULL. > > int A[]; > A[0] = size(A); > > Another example might be > > int A[]; > if (contains(A, 0)) { > A[1] = x; > } > > This example could make progress if the language was smart enough to infer > that A[0] wasn't assigned in the conditional. That's in a sense what confuses me. If we re-write that as: int A0, A1; if (A0 == 1) { A1 = x; } This does, in K, result in a compile-time error. I don't see a distinction between the two cases, and I don't understand why my brain finds it difficult to implement it in the array case. > > I think to better pin down the specification/implementation we probably > need some kind of effect system that models more precisely which bits of > code have what effect on the data structures - > http://en.wikipedia.org/wiki/Effect_system > > We have a prototypical effect system with the write reference counting but > you can have more granular ones where you track which parts of the array > might be modified, etc. Right. Mihael From ketan at mcs.anl.gov Wed Mar 25 10:52:00 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Wed, 25 Mar 2015 10:52:00 -0500 Subject: [Swift-devel] dist/swift-svn to dist/ in git Message-ID: Hi, A bit pedantic but we may want to change the distribution directory from swift-svn to swift-git as we are using git now. Or may be remove one level of dir altogether and have just dist/ to put build targets. -- Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Wed Mar 25 11:56:01 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Wed, 25 Mar 2015 11:56:01 -0500 Subject: [Swift-devel] dist/swift-svn to dist/ in git In-Reply-To: References: Message-ID: We could also change it to something neutral like swift-trunk or swift-vc or swift-devel. - Tim On 25 March 2015 at 10:52, Ketan Maheshwari wrote: > Hi, > > A bit pedantic but we may want to change the distribution directory from > swift-svn to swift-git as we are using git now. Or may be remove one level > of dir altogether and have just dist/ to put build targets. > > -- > Ketan > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Wed Mar 25 16:07:54 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Mar 2015 14:07:54 -0700 Subject: [Swift-devel] At CI tomorrow and Friday Message-ID: <1427317674.4722.0.camel@echo> Hi, As the subject says. I know at least Yadu wanted to talk to me about something. Mihael From tim.g.armstrong at gmail.com Wed Mar 25 16:23:56 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Wed, 25 Mar 2015 16:23:56 -0500 Subject: [Swift-devel] At CI tomorrow and Friday In-Reply-To: <1427317674.4722.0.camel@echo> References: <1427317674.4722.0.camel@echo> Message-ID: What times will you be around? Would be good to catch up? On Mar 25, 2015 4:07 PM, "Mihael Hategan" wrote: > Hi, > > As the subject says. I know at least Yadu wanted to talk to me about > something. > > Mihael > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yadunand at uchicago.edu Wed Mar 25 16:33:40 2015 From: yadunand at uchicago.edu (Yadu Nand Babuji) Date: Wed, 25 Mar 2015 16:33:40 -0500 Subject: [Swift-devel] At CI tomorrow and Friday In-Reply-To: <1427317674.4722.0.camel@echo> References: <1427317674.4722.0.camel@echo> Message-ID: <551329B4.4050609@uchicago.edu> This is great. I have a bunch of stuff that could use your help. -Yadu On 03/25/2015 04:07 PM, Mihael Hategan wrote: > Hi, > > As the subject says. I know at least Yadu wanted to talk to me about > something. > > Mihael > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Wed Mar 25 16:36:53 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Mar 2015 14:36:53 -0700 Subject: [Swift-devel] At CI tomorrow and Friday In-Reply-To: References: <1427317674.4722.0.camel@echo> Message-ID: <1427319413.6010.1.camel@echo> On Wed, 2015-03-25 at 16:23 -0500, Tim Armstrong wrote: > What times will you be around? Would be good to catch up? Hopefully before noon. Yeah, I wanted to talk to you about formalizing the closing semantics. Mihael > On Mar 25, 2015 4:07 PM, "Mihael Hategan" wrote: > > > Hi, > > > > As the subject says. I know at least Yadu wanted to talk to me about > > something. > > > > Mihael > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > From tim.g.armstrong at gmail.com Wed Mar 25 16:41:16 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Wed, 25 Mar 2015 16:41:16 -0500 Subject: [Swift-devel] At CI tomorrow and Friday In-Reply-To: <1427319413.6010.1.camel@echo> References: <1427317674.4722.0.camel@echo> <1427319413.6010.1.camel@echo> Message-ID: Cool, maybe we can meet on Friday afternoon if you're around then? My schedule tomorrow is a bit fragmented. - Tim On 25 March 2015 at 16:36, Mihael Hategan wrote: > On Wed, 2015-03-25 at 16:23 -0500, Tim Armstrong wrote: > > What times will you be around? Would be good to catch up? > > Hopefully before noon. > Yeah, I wanted to talk to you about formalizing the closing semantics. > > Mihael > > > On Mar 25, 2015 4:07 PM, "Mihael Hategan" wrote: > > > > > Hi, > > > > > > As the subject says. I know at least Yadu wanted to talk to me about > > > something. > > > > > > Mihael > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Wed Mar 25 17:23:34 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Mar 2015 15:23:34 -0700 Subject: [Swift-devel] At CI tomorrow and Friday In-Reply-To: References: <1427317674.4722.0.camel@echo> <1427319413.6010.1.camel@echo> Message-ID: <1427322214.6312.0.camel@echo> Sounds good. Mihael On Wed, 2015-03-25 at 16:41 -0500, Tim Armstrong wrote: > Cool, maybe we can meet on Friday afternoon if you're around then? My > schedule tomorrow is a bit fragmented. > > - Tim > > On 25 March 2015 at 16:36, Mihael Hategan wrote: > > > On Wed, 2015-03-25 at 16:23 -0500, Tim Armstrong wrote: > > > What times will you be around? Would be good to catch up? > > > > Hopefully before noon. > > Yeah, I wanted to talk to you about formalizing the closing semantics. > > > > Mihael > > > > > On Mar 25, 2015 4:07 PM, "Mihael Hategan" wrote: > > > > > > > Hi, > > > > > > > > As the subject says. I know at least Yadu wanted to talk to me about > > > > something. > > > > > > > > Mihael > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > > > > > From hategan at mcs.anl.gov Wed Mar 25 22:09:07 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Mar 2015 20:09:07 -0700 Subject: [Swift-devel] assertion behavior Message-ID: <1427339347.8843.0.camel@echo> Hi, In Swift/T, what happens when an assertion fails? Mihael From tim.g.armstrong at gmail.com Wed Mar 25 22:24:59 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Wed, 25 Mar 2015 22:24:59 -0500 Subject: [Swift-devel] assertion behavior In-Reply-To: <1427339347.8843.0.camel@echo> References: <1427339347.8843.0.camel@echo> Message-ID: The program terminates immediately with an error message. There's a compile-time option to disable assertions if needed. It's actually a little weird in that it syntactically removes the statement - it doesn't evaluate the arguments to the funciton. - Tim On 25 March 2015 at 22:09, Mihael Hategan wrote: > Hi, > > In Swift/T, what happens when an assertion fails? > > Mihael > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Fri Mar 27 09:20:49 2015 From: wilde at anl.gov (Michael Wilde) Date: Fri, 27 Mar 2015 09:20:49 -0500 Subject: [Swift-devel] assertion behavior In-Reply-To: References: <1427339347.8843.0.camel@echo> Message-ID: <55156741.6060901@anl.gov> Does "terminate immediately" need to always be immediate or should/could it integrate with the Swift/K notion of "soft errors"? I.e. an assert failure is treated like a failing function; that in turn as handled as the soft-error property specifies. - Mike On 3/25/15 10:24 PM, Tim Armstrong wrote: > The program terminates immediately with an error message. > > There's a compile-time option to disable assertions if needed. It's > actually a little weird in that it syntactically removes the statement > - it doesn't evaluate the arguments to the funciton. > > - Tim > > On 25 March 2015 at 22:09, Mihael Hategan > wrote: > > Hi, > > In Swift/T, what happens when an assertion fails? > > Mihael > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Fri Mar 27 09:39:56 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 27 Mar 2015 09:39:56 -0500 Subject: [Swift-devel] assertion behavior In-Reply-To: <55156741.6060901@anl.gov> References: <1427339347.8843.0.camel@echo> <55156741.6060901@anl.gov> Message-ID: I don't think it makes sense to treat it as a soft error. The primary use case for assertions is the Swift/T test suite and there's no reason to continue running after an assertion failure. If a user wants to use them in their own scripts, they're intended to catch bugs or unintended behaviour. If an assertion fails, the script is not doing what it's meant to be doing. I don't see any reason to keep on running a buggy script that failed an assertion, and I see plenty of reasons not to keep it running. If users want to disable their assertions there is support for that. - Tim On 27 March 2015 at 09:20, Michael Wilde wrote: > Does "terminate immediately" need to always be immediate or should/could > it integrate with the Swift/K notion of "soft errors"? > > I.e. an assert failure is treated like a failing function; that in turn as > handled as the soft-error property specifies. > - Mike > > > > On 3/25/15 10:24 PM, Tim Armstrong wrote: > > The program terminates immediately with an error message. > > There's a compile-time option to disable assertions if needed. It's > actually a little weird in that it syntactically removes the statement - it > doesn't evaluate the arguments to the funciton. > > - Tim > > On 25 March 2015 at 22:09, Mihael Hategan wrote: > >> Hi, >> >> In Swift/T, what happens when an assertion fails? >> >> Mihael >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> > > > > _______________________________________________ > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wozniak at mcs.anl.gov Fri Mar 27 12:28:07 2015 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Fri, 27 Mar 2015 12:28:07 -0500 Subject: [Swift-devel] assertion behavior In-Reply-To: References: <1427339347.8843.0.camel@echo> <55156741.6060901@anl.gov> Message-ID: <55159327.3090607@mcs.anl.gov> Catchable/soft assertion failures would be useful if we rewrote the test suite in Swift. Maybe we should do that anyway. On 3/27/2015 9:39 AM, Tim Armstrong wrote: > I don't think it makes sense to treat it as a soft error. > > The primary use case for assertions is the Swift/T test suite and > there's no reason to continue running after an assertion failure. > > If a user wants to use them in their own scripts, they're intended to > catch bugs or unintended behaviour. If an assertion fails, the script > is not doing what it's meant to be doing. I don't see any reason to > keep on running a buggy script that failed an assertion, and I see > plenty of reasons not to keep it running. > > If users want to disable their assertions there is support for that. > > - Tim > > On 27 March 2015 at 09:20, Michael Wilde > wrote: > > Does "terminate immediately" need to always be immediate or > should/could it integrate with the Swift/K notion of "soft errors"? > > I.e. an assert failure is treated like a failing function; that in > turn as handled as the soft-error property specifies. > - Mike > > > > On 3/25/15 10:24 PM, Tim Armstrong wrote: >> The program terminates immediately with an error message. >> >> There's a compile-time option to disable assertions if needed. >> It's actually a little weird in that it syntactically removes the >> statement - it doesn't evaluate the arguments to the funciton. >> >> - Tim >> >> On 25 March 2015 at 22:09, Mihael Hategan > > wrote: >> >> Hi, >> >> In Swift/T, what happens when an assertion fails? >> >> Mihael >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> >> >> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Justin M Wozniak -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Fri Mar 27 21:58:44 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 27 Mar 2015 21:58:44 -0500 Subject: [Swift-devel] assertion behavior In-Reply-To: <55159327.3090607@mcs.anl.gov> References: <1427339347.8843.0.camel@echo> <55156741.6060901@anl.gov> <55159327.3090607@mcs.anl.gov> Message-ID: I feel like that is getting away from what assertions are intended to be - the best thing about them is that they're dead simple and have a clear, well-defined use case. I don't think that should be compromised out of a desire to have them serve dual purposes. - Tim On 27 March 2015 at 12:28, Justin M Wozniak wrote: > > Catchable/soft assertion failures would be useful if we rewrote the test > suite in Swift. Maybe we should do that anyway. > > On 3/27/2015 9:39 AM, Tim Armstrong wrote: > > I don't think it makes sense to treat it as a soft error. > > The primary use case for assertions is the Swift/T test suite and there's > no reason to continue running after an assertion failure. > > If a user wants to use them in their own scripts, they're intended to > catch bugs or unintended behaviour. If an assertion fails, the script is > not doing what it's meant to be doing. I don't see any reason to keep on > running a buggy script that failed an assertion, and I see plenty of > reasons not to keep it running. > > If users want to disable their assertions there is support for that. > > - Tim > > On 27 March 2015 at 09:20, Michael Wilde wrote: > >> Does "terminate immediately" need to always be immediate or should/could >> it integrate with the Swift/K notion of "soft errors"? >> >> I.e. an assert failure is treated like a failing function; that in turn >> as handled as the soft-error property specifies. >> - Mike >> >> >> >> On 3/25/15 10:24 PM, Tim Armstrong wrote: >> >> The program terminates immediately with an error message. >> >> There's a compile-time option to disable assertions if needed. It's >> actually a little weird in that it syntactically removes the statement - it >> doesn't evaluate the arguments to the funciton. >> >> - Tim >> >> On 25 March 2015 at 22:09, Mihael Hategan wrote: >> >>> Hi, >>> >>> In Swift/T, what happens when an assertion fails? >>> >>> Mihael >>> >>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>> >> >> >> >> _______________________________________________ >> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> >> >> -- >> Michael Wilde >> Mathematics and Computer Science Computation Institute >> Argonne National Laboratory The University of Chicago >> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> >> > > > _______________________________________________ > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > -- > Justin M Wozniak > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sat Mar 28 00:43:32 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 27 Mar 2015 22:43:32 -0700 Subject: [Swift-devel] assertion behavior In-Reply-To: References: <1427339347.8843.0.camel@echo> <55156741.6060901@anl.gov> <55159327.3090607@mcs.anl.gov> Message-ID: <1427521412.11240.1.camel@echo> Right. I also think that a hard abort is the right choice. An assertion is a fundamental statement about program correctness. If that fails, there shouldn't be much of a point in continuing. In other news, we still need some kind of fault isolation and handling mechanism. Mihael On Fri, 2015-03-27 at 21:58 -0500, Tim Armstrong wrote: > I feel like that is getting away from what assertions are intended to be - > the best thing about them is that they're dead simple and have a clear, > well-defined use case. I don't think that should be compromised out of a > desire to have them serve dual purposes. > > - Tim > > On 27 March 2015 at 12:28, Justin M Wozniak wrote: > > > > > Catchable/soft assertion failures would be useful if we rewrote the test > > suite in Swift. Maybe we should do that anyway. > > > > On 3/27/2015 9:39 AM, Tim Armstrong wrote: > > > > I don't think it makes sense to treat it as a soft error. > > > > The primary use case for assertions is the Swift/T test suite and there's > > no reason to continue running after an assertion failure. > > > > If a user wants to use them in their own scripts, they're intended to > > catch bugs or unintended behaviour. If an assertion fails, the script is > > not doing what it's meant to be doing. I don't see any reason to keep on > > running a buggy script that failed an assertion, and I see plenty of > > reasons not to keep it running. > > > > If users want to disable their assertions there is support for that. > > > > - Tim > > > > On 27 March 2015 at 09:20, Michael Wilde wrote: > > > >> Does "terminate immediately" need to always be immediate or should/could > >> it integrate with the Swift/K notion of "soft errors"? > >> > >> I.e. an assert failure is treated like a failing function; that in turn > >> as handled as the soft-error property specifies. > >> - Mike > >> > >> > >> > >> On 3/25/15 10:24 PM, Tim Armstrong wrote: > >> > >> The program terminates immediately with an error message. > >> > >> There's a compile-time option to disable assertions if needed. It's > >> actually a little weird in that it syntactically removes the statement - it > >> doesn't evaluate the arguments to the funciton. > >> > >> - Tim > >> > >> On 25 March 2015 at 22:09, Mihael Hategan wrote: > >> > >>> Hi, > >>> > >>> In Swift/T, what happens when an assertion fails? > >>> > >>> Mihael > >>> > >>> _______________________________________________ > >>> Swift-devel mailing list > >>> Swift-devel at ci.uchicago.edu > >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > >>> > >> > >> > >> > >> _______________________________________________ > >> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > >> > >> > >> -- > >> Michael Wilde > >> Mathematics and Computer Science Computation Institute > >> Argonne National Laboratory The University of Chicago > >> > >> > >> _______________________________________________ > >> Swift-devel mailing list > >> Swift-devel at ci.uchicago.edu > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > >> > >> > > > > > > _______________________________________________ > > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > > > -- > > Justin M Wozniak > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Sun Mar 29 20:01:01 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 29 Mar 2015 18:01:01 -0700 Subject: [Swift-devel] documentation formatting Message-ID: <1427677261.21345.2.camel@echo> Hi, I played a bit with the format of our docs. You can see the results at http://www.mcs.anl.gov/~hategan/userguide/userguide.html I've tested it on Firefox and Chrome. It might not work as well on other browser, but the point is to get feedback on the general idea. Mihael From wilde at mcs.anl.gov Sun Mar 29 20:15:47 2015 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sun, 29 Mar 2015 20:15:47 -0500 Subject: [Swift-devel] documentation formatting In-Reply-To: <55c69c1410434e9abe36289486f39844@GEORGE.anl.gov> References: <55c69c1410434e9abe36289486f39844@GEORGE.anl.gov> Message-ID: Nice! I like it. :) Is it generated from the AsciiDoc source in an automated (or automatable) manner? A few thoughts: - would be great if the left column menu was a collapsable/expandable outline (eventually) - would be better if the right side was in the current single-doc format, at least while its of manageable size. Makes it easier to search. - scroll behavior is a bit odd: when the left menu hits the top or bottom, the right-column text start scrolling. That might be desirable, not sure. But it felt a tad strange. Regardless, I think this is great and a nice improvement in readability. Thanks for suggesting it and doing it. We should discuss whats needed to make this the new live format - or at least an optional link users can select on the doc page (e.g., "New indexed User Guide") - Mike On Sun, Mar 29, 2015 at 8:01 PM, Hategan-Marandiuc, Philip M. < hategan at mcs.anl.gov> wrote: > Hi, > > I played a bit with the format of our docs. You can see the results at > http://www.mcs.anl.gov/~hategan/userguide/userguide.html > > I've tested it on Firefox and Chrome. It might not work as well on other > browser, but the point is to get feedback on the general idea. > > Mihael > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Sun Mar 29 20:51:56 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Sun, 29 Mar 2015 20:51:56 -0500 Subject: [Swift-devel] documentation formatting In-Reply-To: References: <55c69c1410434e9abe36289486f39844@GEORGE.anl.gov> Message-ID: Two thumbs up. I think I prefer the left as it is without any collapsing/expanding - I personally find expanding/collapsing outlines are overly fiddly - you have to click multiple times to get to a section. One problem I see is that it doesn't scale down to below a certain screen width - it might not work well on devices with smaller screens. The code examples impose a minimum width to some extend, but it would be good if it scaled a little smaller. Anyway, a minor concern - it's a big improvement. - Tim On 29 March 2015 at 20:15, Michael Wilde wrote: > Nice! I like it. :) > > Is it generated from the AsciiDoc source in an automated (or automatable) > manner? > > A few thoughts: > > - would be great if the left column menu was a collapsable/expandable > outline (eventually) > > - would be better if the right side was in the current single-doc format, > at least while its of manageable size. Makes it easier to search. > > - scroll behavior is a bit odd: when the left menu hits the top or bottom, > the right-column text start scrolling. That might be desirable, not sure. > But it felt a tad strange. > > Regardless, I think this is great and a nice improvement in readability. > Thanks for suggesting it and doing it. > > We should discuss whats needed to make this the new live format - or at > least an optional link users can select on the doc page (e.g., "New indexed > User Guide") > > - Mike > > > On Sun, Mar 29, 2015 at 8:01 PM, Hategan-Marandiuc, Philip M. < > hategan at mcs.anl.gov> wrote: > >> Hi, >> >> I played a bit with the format of our docs. You can see the results at >> http://www.mcs.anl.gov/~hategan/userguide/userguide.html >> >> I've tested it on Firefox and Chrome. It might not work as well on other >> browser, but the point is to get feedback on the general idea. >> >> Mihael >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Sun Mar 29 20:58:21 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Sun, 29 Mar 2015 20:58:21 -0500 Subject: [Swift-devel] documentation formatting In-Reply-To: References: <55c69c1410434e9abe36289486f39844@GEORGE.anl.gov> Message-ID: This looks great! May be we want to utilize some space on the side so that wider tables/code can be nicely accommodated. A footer strip (with links, license info etc.) similar to the header one might also look nice. -- Ketan On Sun, Mar 29, 2015 at 8:51 PM, Tim Armstrong wrote: > Two thumbs up. > > I think I prefer the left as it is without any collapsing/expanding - I > personally find expanding/collapsing outlines are overly fiddly - you have > to click multiple times to get to a section. > > One problem I see is that it doesn't scale down to below a certain screen > width - it might not work well on devices with smaller screens. The code > examples impose a minimum width to some extend, but it would be good if it > scaled a little smaller. Anyway, a minor concern - it's a big improvement. > > - Tim > > On 29 March 2015 at 20:15, Michael Wilde wrote: > >> Nice! I like it. :) >> >> Is it generated from the AsciiDoc source in an automated (or automatable) >> manner? >> >> A few thoughts: >> >> - would be great if the left column menu was a collapsable/expandable >> outline (eventually) >> >> - would be better if the right side was in the current single-doc format, >> at least while its of manageable size. Makes it easier to search. >> >> - scroll behavior is a bit odd: when the left menu hits the top or >> bottom, the right-column text start scrolling. That might be desirable, not >> sure. But it felt a tad strange. >> >> Regardless, I think this is great and a nice improvement in readability. >> Thanks for suggesting it and doing it. >> >> We should discuss whats needed to make this the new live format - or at >> least an optional link users can select on the doc page (e.g., "New indexed >> User Guide") >> >> - Mike >> >> >> On Sun, Mar 29, 2015 at 8:01 PM, Hategan-Marandiuc, Philip M. < >> hategan at mcs.anl.gov> wrote: >> >>> Hi, >>> >>> I played a bit with the format of our docs. You can see the results at >>> http://www.mcs.anl.gov/~hategan/userguide/userguide.html >>> >>> I've tested it on Firefox and Chrome. It might not work as well on other >>> browser, but the point is to get feedback on the general idea. >>> >>> Mihael >>> >>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>> >> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> >> > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sun Mar 29 21:58:10 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 29 Mar 2015 19:58:10 -0700 Subject: [Swift-devel] documentation formatting In-Reply-To: References: <55c69c1410434e9abe36289486f39844@GEORGE.anl.gov> Message-ID: <1427684290.21345.8.camel@echo> On Sun, 2015-03-29 at 20:15 -0500, Michael Wilde wrote: > Nice! I like it. :) > > Is it generated from the AsciiDoc source in an automated (or automatable) > manner? Yes. It's the existing asciidoc stuff with some additional stylesheets and some javascript. It's compiled the exact same way as our current docs are. > > A few thoughts: > > - would be great if the left column menu was a collapsable/expandable > outline (eventually) I agree. > > - would be better if the right side was in the current single-doc format, > at least while its of manageable size. Makes it easier to search. We could have a search function. It is actually one html page. The swapping of sections is done in javascript. > > - scroll behavior is a bit odd: when the left menu hits the top or bottom, > the right-column text start scrolling. That might be desirable, not sure. > But it felt a tad strange. What browser? I don't think I've seen that. > > Regardless, I think this is great and a nice improvement in readability. > Thanks for suggesting it and doing it. > > We should discuss whats needed to make this the new live format - or at > least an optional link users can select on the doc page (e.g., "New indexed > User Guide") It's one commit away really, but we should probably do this in a branch as part of 0.97 documentation efforts. Mihael From hategan at mcs.anl.gov Sun Mar 29 22:00:36 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 29 Mar 2015 20:00:36 -0700 Subject: [Swift-devel] documentation formatting In-Reply-To: References: <55c69c1410434e9abe36289486f39844@GEORGE.anl.gov> Message-ID: <1427684436.21345.10.camel@echo> On Sun, 2015-03-29 at 20:51 -0500, Tim Armstrong wrote: > Two thumbs up. > > I think I prefer the left as it is without any collapsing/expanding - I > personally find expanding/collapsing outlines are overly fiddly - you have > to click multiple times to get to a section. > > One problem I see is that it doesn't scale down to below a certain screen > width - it might not work well on devices with smaller screens. The code > examples impose a minimum width to some extend, but it would be good if it > scaled a little smaller. Anyway, a minor concern - it's a big improvement. It might be doable so that it's reasonably wide on large screens and still fits on smaller ones, but I'm no web stuff expert. Mihael From hategan at mcs.anl.gov Sun Mar 29 22:04:03 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 29 Mar 2015 20:04:03 -0700 Subject: [Swift-devel] documentation formatting In-Reply-To: References: <55c69c1410434e9abe36289486f39844@GEORGE.anl.gov> Message-ID: <1427684643.21345.14.camel@echo> On Sun, 2015-03-29 at 20:58 -0500, Ketan Maheshwari wrote: > May be we want to utilize some space on the side so that wider tables/code > can be nicely accommodated. It is possible that I might have gotten inspired by the documentation of a very similarly named language by a relatively large company. It might have used the same size. There's probably something in the usability manual about very long lines being hard to read. > A footer strip (with links, license info etc.) > similar to the header one might also look nice. Right. We should probably run this by somebody with experience in web design before we call it final. Mihael From lpesce at uchicago.edu Mon Mar 30 09:51:36 2015 From: lpesce at uchicago.edu (Lorenzo Pesce) Date: Mon, 30 Mar 2015 09:51:36 -0500 Subject: [Swift-devel] documentation formatting In-Reply-To: <1427677261.21345.2.camel@echo> References: <1427677261.21345.2.camel@echo> Message-ID: <26DDCE2D-2C0B-47FA-9842-54C65850A78E@uchicago.edu> For what my opinion is worth, I like it and find it easy to navigate. > On Mar 29, 2015, at 8:01 PM, Mihael Hategan wrote: > > Hi, > > I played a bit with the format of our docs. You can see the results at > http://www.mcs.anl.gov/~hategan/userguide/userguide.html > > I've tested it on Firefox and Chrome. It might not work as well on other > browser, but the point is to get feedback on the general idea. > > Mihael > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From wozniak at mcs.anl.gov Mon Mar 30 10:03:17 2015 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Mon, 30 Mar 2015 10:03:17 -0500 Subject: [Swift-devel] documentation formatting In-Reply-To: <1427684436.21345.10.camel@echo> References: <55c69c1410434e9abe36289486f39844@GEORGE.anl.gov> <1427684436.21345.10.camel@echo> Message-ID: <551965B5.6000801@mcs.anl.gov> On 03/29/2015 10:00 PM, Mihael Hategan wrote: >> >One problem I see is that it doesn't scale down to below a certain screen >> >width - it might not work well on devices with smaller screens. The code >> >examples impose a minimum width to some extend, but it would be good if it >> >scaled a little smaller. Anyway, a minor concern - it's a big improvement. > It might be doable so that it's reasonably wide on large screens and > still fits on smaller ones, but I'm no web stuff expert. I will try to apply this to the Swift/T docs and set it up so the user can choose a version without frames. -- Justin M Wozniak From wozniak at mcs.anl.gov Tue Mar 31 10:25:09 2015 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Tue, 31 Mar 2015 10:25:09 -0500 Subject: [Swift-devel] assertion behavior In-Reply-To: <1427521412.11240.1.camel@echo> References: <1427339347.8843.0.camel@echo> <55156741.6060901@anl.gov> <55159327.3090607@mcs.anl.gov> <1427521412.11240.1.camel@echo> Message-ID: <551ABC55.5050308@mcs.anl.gov> If assert failure always returns control to the OS, should we use some other symbol in the test suite? Or is managing testable code from Swift a bad idea? (This isn't just about core Swift tests, this is about supporting test-driven development in Swift.) On 03/28/2015 12:43 AM, Mihael Hategan wrote: > Right. I also think that a hard abort is the right choice. An assertion > is a fundamental statement about program correctness. If that fails, > there shouldn't be much of a point in continuing. > > In other news, we still need some kind of fault isolation and handling > mechanism. > > Mihael > > On Fri, 2015-03-27 at 21:58 -0500, Tim Armstrong wrote: >> I feel like that is getting away from what assertions are intended to be - >> the best thing about them is that they're dead simple and have a clear, >> well-defined use case. I don't think that should be compromised out of a >> desire to have them serve dual purposes. >> >> - Tim >> >> On 27 March 2015 at 12:28, Justin M Wozniak wrote: >> >>> Catchable/soft assertion failures would be useful if we rewrote the test >>> suite in Swift. Maybe we should do that anyway. >>> >>> On 3/27/2015 9:39 AM, Tim Armstrong wrote: >>> >>> I don't think it makes sense to treat it as a soft error. >>> >>> The primary use case for assertions is the Swift/T test suite and there's >>> no reason to continue running after an assertion failure. >>> >>> If a user wants to use them in their own scripts, they're intended to >>> catch bugs or unintended behaviour. If an assertion fails, the script is >>> not doing what it's meant to be doing. I don't see any reason to keep on >>> running a buggy script that failed an assertion, and I see plenty of >>> reasons not to keep it running. >>> >>> If users want to disable their assertions there is support for that. >>> >>> - Tim >>> >>> On 27 March 2015 at 09:20, Michael Wilde wrote: >>> >>>> Does "terminate immediately" need to always be immediate or should/could >>>> it integrate with the Swift/K notion of "soft errors"? >>>> >>>> I.e. an assert failure is treated like a failing function; that in turn >>>> as handled as the soft-error property specifies. >>>> - Mike >>>> >>>> >>>> >>>> On 3/25/15 10:24 PM, Tim Armstrong wrote: >>>> >>>> The program terminates immediately with an error message. >>>> >>>> There's a compile-time option to disable assertions if needed. It's >>>> actually a little weird in that it syntactically removes the statement - it >>>> doesn't evaluate the arguments to the funciton. >>>> >>>> - Tim >>>> >>>> On 25 March 2015 at 22:09, Mihael Hategan wrote: >>>> >>>>> Hi, >>>>> >>>>> In Swift/T, what happens when an assertion fails? >>>>> >>>>> Mihael >>>>> >>>>> _______________________________________________ >>>>> Swift-devel mailing list >>>>> Swift-devel at ci.uchicago.edu >>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>> >>>> >>>> -- >>>> Michael Wilde >>>> Mathematics and Computer Science Computation Institute >>>> Argonne National Laboratory The University of Chicago >>>> >>>> >>>> _______________________________________________ >>>> Swift-devel mailing list >>>> Swift-devel at ci.uchicago.edu >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>> >>>> >>> >>> _______________________________________________ >>> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>> >>> >>> >>> -- >>> Justin M Wozniak >>> >>> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -- Justin M Wozniak From tim.g.armstrong at gmail.com Tue Mar 31 11:30:07 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Tue, 31 Mar 2015 11:30:07 -0500 Subject: [Swift-devel] assertion behavior In-Reply-To: <551ABC55.5050308@mcs.anl.gov> References: <1427339347.8843.0.camel@echo> <55156741.6060901@anl.gov> <55159327.3090607@mcs.anl.gov> <1427521412.11240.1.camel@echo> <551ABC55.5050308@mcs.anl.gov> Message-ID: I think Swift application testing has fairly different requirements from what traditional assertions provide. Hard aborts work well for the language testing since if there are a language-level bugs we have no idea what state things are left in from test to test. They also work well for catching logic errors in programs (the classic use case for asserts). I think if you want to build SwiftUnit (or whatever it would be called), it would probably need to use different mechanisms from the regular asserts. - Tim On 31 March 2015 at 10:25, Justin M Wozniak wrote: > > If assert failure always returns control to the OS, should we use some > other symbol in the test suite? Or is managing testable code from Swift a > bad idea? (This isn't just about core Swift tests, this is about > supporting test-driven development in Swift.) > > > On 03/28/2015 12:43 AM, Mihael Hategan wrote: > >> Right. I also think that a hard abort is the right choice. An assertion >> is a fundamental statement about program correctness. If that fails, >> there shouldn't be much of a point in continuing. >> >> In other news, we still need some kind of fault isolation and handling >> mechanism. >> >> Mihael >> >> On Fri, 2015-03-27 at 21:58 -0500, Tim Armstrong wrote: >> >>> I feel like that is getting away from what assertions are intended to be >>> - >>> the best thing about them is that they're dead simple and have a clear, >>> well-defined use case. I don't think that should be compromised out of a >>> desire to have them serve dual purposes. >>> >>> - Tim >>> >>> On 27 March 2015 at 12:28, Justin M Wozniak wrote: >>> >>> Catchable/soft assertion failures would be useful if we rewrote the test >>>> suite in Swift. Maybe we should do that anyway. >>>> >>>> On 3/27/2015 9:39 AM, Tim Armstrong wrote: >>>> >>>> I don't think it makes sense to treat it as a soft error. >>>> >>>> The primary use case for assertions is the Swift/T test suite and >>>> there's >>>> no reason to continue running after an assertion failure. >>>> >>>> If a user wants to use them in their own scripts, they're intended to >>>> catch bugs or unintended behaviour. If an assertion fails, the script >>>> is >>>> not doing what it's meant to be doing. I don't see any reason to keep >>>> on >>>> running a buggy script that failed an assertion, and I see plenty of >>>> reasons not to keep it running. >>>> >>>> If users want to disable their assertions there is support for that. >>>> >>>> - Tim >>>> >>>> On 27 March 2015 at 09:20, Michael Wilde wrote: >>>> >>>> Does "terminate immediately" need to always be immediate or >>>>> should/could >>>>> it integrate with the Swift/K notion of "soft errors"? >>>>> >>>>> I.e. an assert failure is treated like a failing function; that in turn >>>>> as handled as the soft-error property specifies. >>>>> - Mike >>>>> >>>>> >>>>> >>>>> On 3/25/15 10:24 PM, Tim Armstrong wrote: >>>>> >>>>> The program terminates immediately with an error message. >>>>> >>>>> There's a compile-time option to disable assertions if needed. It's >>>>> actually a little weird in that it syntactically removes the statement >>>>> - it >>>>> doesn't evaluate the arguments to the funciton. >>>>> >>>>> - Tim >>>>> >>>>> On 25 March 2015 at 22:09, Mihael Hategan wrote: >>>>> >>>>> Hi, >>>>>> >>>>>> In Swift/T, what happens when an assertion fails? >>>>>> >>>>>> Mihael >>>>>> >>>>>> _______________________________________________ >>>>>> Swift-devel mailing list >>>>>> Swift-devel at ci.uchicago.edu >>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps:// >>>>> lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>>> >>>>> >>>>> -- >>>>> Michael Wilde >>>>> Mathematics and Computer Science Computation Institute >>>>> Argonne National Laboratory The University of Chicago >>>>> >>>>> >>>>> _______________________________________________ >>>>> Swift-devel mailing list >>>>> Swift-devel at ci.uchicago.edu >>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps:// >>>> lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>> >>>> >>>> >>>> -- >>>> Justin M Wozniak >>>> >>>> >>>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>> >> >> > > -- > Justin M Wozniak > > On 31 March 2015 at 10:25, Justin M Wozniak wrote: > > If assert failure always returns control to the OS, should we use some > other symbol in the test suite? Or is managing testable code from Swift a > bad idea? (This isn't just about core Swift tests, this is about > supporting test-driven development in Swift.) > > > On 03/28/2015 12:43 AM, Mihael Hategan wrote: > >> Right. I also think that a hard abort is the right choice. An assertion >> is a fundamental statement about program correctness. If that fails, >> there shouldn't be much of a point in continuing. >> >> In other news, we still need some kind of fault isolation and handling >> mechanism. >> >> Mihael >> >> On Fri, 2015-03-27 at 21:58 -0500, Tim Armstrong wrote: >> >>> I feel like that is getting away from what assertions are intended to be >>> - >>> the best thing about them is that they're dead simple and have a clear, >>> well-defined use case. I don't think that should be compromised out of a >>> desire to have them serve dual purposes. >>> >>> - Tim >>> >>> On 27 March 2015 at 12:28, Justin M Wozniak wrote: >>> >>> Catchable/soft assertion failures would be useful if we rewrote the test >>>> suite in Swift. Maybe we should do that anyway. >>>> >>>> On 3/27/2015 9:39 AM, Tim Armstrong wrote: >>>> >>>> I don't think it makes sense to treat it as a soft error. >>>> >>>> The primary use case for assertions is the Swift/T test suite and >>>> there's >>>> no reason to continue running after an assertion failure. >>>> >>>> If a user wants to use them in their own scripts, they're intended to >>>> catch bugs or unintended behaviour. If an assertion fails, the script >>>> is >>>> not doing what it's meant to be doing. I don't see any reason to keep >>>> on >>>> running a buggy script that failed an assertion, and I see plenty of >>>> reasons not to keep it running. >>>> >>>> If users want to disable their assertions there is support for that. >>>> >>>> - Tim >>>> >>>> On 27 March 2015 at 09:20, Michael Wilde wrote: >>>> >>>> Does "terminate immediately" need to always be immediate or >>>>> should/could >>>>> it integrate with the Swift/K notion of "soft errors"? >>>>> >>>>> I.e. an assert failure is treated like a failing function; that in turn >>>>> as handled as the soft-error property specifies. >>>>> - Mike >>>>> >>>>> >>>>> >>>>> On 3/25/15 10:24 PM, Tim Armstrong wrote: >>>>> >>>>> The program terminates immediately with an error message. >>>>> >>>>> There's a compile-time option to disable assertions if needed. It's >>>>> actually a little weird in that it syntactically removes the statement >>>>> - it >>>>> doesn't evaluate the arguments to the funciton. >>>>> >>>>> - Tim >>>>> >>>>> On 25 March 2015 at 22:09, Mihael Hategan wrote: >>>>> >>>>> Hi, >>>>>> >>>>>> In Swift/T, what happens when an assertion fails? >>>>>> >>>>>> Mihael >>>>>> >>>>>> _______________________________________________ >>>>>> Swift-devel mailing list >>>>>> Swift-devel at ci.uchicago.edu >>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps:// >>>>> lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>>> >>>>> >>>>> -- >>>>> Michael Wilde >>>>> Mathematics and Computer Science Computation Institute >>>>> Argonne National Laboratory The University of Chicago >>>>> >>>>> >>>>> _______________________________________________ >>>>> Swift-devel mailing list >>>>> Swift-devel at ci.uchicago.edu >>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps:// >>>> lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>>> >>>> >>>> >>>> -- >>>> Justin M Wozniak >>>> >>>> >>>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >>> >> >> > > -- > Justin M Wozniak > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Tue Mar 31 16:16:08 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Tue, 31 Mar 2015 16:16:08 -0500 Subject: [Swift-devel] adding a flag to cobalt qsub Message-ID: Hi, To run bootable subjobs, a custom flag "--disable_preboot" needs to be added to the qsub command line of cobalt provider. Can this be done from the config file in trunk? Thanks -- Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Mar 31 16:22:29 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 31 Mar 2015 14:22:29 -0700 Subject: [Swift-devel] adding a flag to cobalt qsub In-Reply-To: References: Message-ID: <1427836949.14105.0.camel@echo> Hi, This is done in CobaltExecutor.buildCommandLine() Mihael On Tue, 2015-03-31 at 16:16 -0500, Ketan Maheshwari wrote: > Hi, > > To run bootable subjobs, a custom flag "--disable_preboot" needs to be > added to the qsub command line of cobalt provider. Can this be done from > the config file in trunk? > > Thanks > -- > Ketan > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel