From ketan at mcs.anl.gov Mon Aug 4 15:17:13 2014 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Mon, 4 Aug 2014 15:17:13 -0500 Subject: [Swift-user] swift.properties overriding commandline parameters Message-ID: Hi, While using Swift 0.95 RC6, I find that the sites.file property is being picked up from my ~/.swift/swift.properties file despite being provided explicitly at command-line. Is this expected behavior? My intuition was that parameters specified on command-line would have highest precedence. Thanks, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jozik at uchicago.edu Fri Aug 8 14:36:26 2014 From: jozik at uchicago.edu (Jonathan Ozik) Date: Fri, 8 Aug 2014 14:36:26 -0500 Subject: [Swift-user] Newer sites properties Message-ID: <235AE1C5-4120-49E4-9B1C-CE786D29555B@uchicago.edu> Is there a place where the newer (0.95+) sites properties are defined? For example, jobWalltime, taskWalltime . I couldn?t find them in the user guide (http://swift-lang.org/guides/trunk/userguide/userguide.html). Thanks! Jonathan From yadudoc1729 at gmail.com Fri Aug 8 16:40:33 2014 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Fri, 8 Aug 2014 16:40:33 -0500 Subject: [Swift-user] Newer sites properties In-Reply-To: <235AE1C5-4120-49E4-9B1C-CE786D29555B@uchicago.edu> References: <235AE1C5-4120-49E4-9B1C-CE786D29555B@uchicago.edu> Message-ID: Hi Jonathan, ?? The 0.95 sites properties were updated with the configs that will be supported on trunk and 0.96. I'm attaching an older userguide which has the configs for swift-0.95. ?Thanks, Yadu? On Fri, Aug 8, 2014 at 2:36 PM, Jonathan Ozik wrote: > Is there a place where the newer (0.95+) sites properties are defined? For > example, jobWalltime, taskWalltime . I couldn?t find them in the user guide > (http://swift-lang.org/guides/trunk/userguide/userguide.html). > > Thanks! > > Jonathan > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -- Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0.95_userguide.pdf Type: application/pdf Size: 393827 bytes Desc: not available URL: From jozik at uchicago.edu Tue Aug 12 11:53:08 2014 From: jozik at uchicago.edu (Jonathan Ozik) Date: Tue, 12 Aug 2014 11:53:08 -0500 Subject: [Swift-user] Swift and allocations Message-ID: Hello all, I had a discussion with Yadu and we thought we?d open it up to the list. Suppose that we are throttling the total number of concurrent tasks to 512 (taskThrottle=512) and by specifying that maxJobs=32 and tasksPerWorker=16. If there are a total of 2430 total tasks to run, there will be 2430 - 2048 = 382 tasks left for the last ?go around? (assuming that each of the tasks take about the same time to complete). If there are 382 tasks left towards run completion and 512 cores (16 for each of the 32 nodes), would the last few jobs, end up spread across the 32 Nodes, effectively holding them and burning your allocation, or would they get pushed to 24 nodes and de-allocating 8 nodes? Jonathan From hategan at mcs.anl.gov Tue Aug 12 12:37:18 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 12 Aug 2014 10:37:18 -0700 Subject: [Swift-user] Swift and allocations In-Reply-To: References: Message-ID: <1407865038.12289.3.camel@echo> Hi Jonathan, There is no specific code to attempt to bunch the invocations to fewer blocks. Essentially the 382 cores that finish their previous 512 jobs first will get the new jobs. However, if any of the nodes are left without jobs, they will get shut down. Mihael On Tue, 2014-08-12 at 11:53 -0500, Jonathan Ozik wrote: > Hello all, > > I had a discussion with Yadu and we thought we?d open it up to the > list. Suppose that we are throttling the total number of concurrent > tasks to 512 (taskThrottle=512) and by specifying that maxJobs=32 and > tasksPerWorker=16. If there are a total of 2430 total tasks to run, > there will be 2430 - 2048 = 382 tasks left for the last ?go > around? (assuming that each of the tasks take about the same time to > complete). If there are 382 tasks left towards run completion and 512 > cores (16 for each of the 32 nodes), would the last few jobs, end up > spread across the 32 Nodes, effectively holding them and burning your > allocation, or would they get pushed to 24 nodes and de-allocating 8 > nodes? > > Jonathan > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From jozik at uchicago.edu Tue Aug 12 13:45:52 2014 From: jozik at uchicago.edu (Jonathan Ozik) Date: Tue, 12 Aug 2014 13:45:52 -0500 Subject: [Swift-user] Swift and allocations In-Reply-To: <1407865038.12289.3.camel@echo> References: <1407865038.12289.3.camel@echo> Message-ID: <0443760C-DE97-4DD9-859E-A3B62DC9584B@uchicago.edu> Hi Mihael, Thank you, good to know. The jobWalltime, should that be thought of as the length of time that each block will stay up? For example, if resources are tight and only a few blocks are started, while the rest of the blocks are waiting for resources to free up, if the first few blocks reach their jobWalltimes, will those be shut down? Jonathan On Aug 12, 2014, at 12:37 PM, Mihael Hategan wrote: > Hi Jonathan, > > There is no specific code to attempt to bunch the invocations to fewer > blocks. Essentially the 382 cores that finish their previous 512 jobs > first will get the new jobs. > > However, if any of the nodes are left without jobs, they will get shut > down. > > Mihael > > On Tue, 2014-08-12 at 11:53 -0500, Jonathan Ozik wrote: >> Hello all, >> >> I had a discussion with Yadu and we thought we?d open it up to the >> list. Suppose that we are throttling the total number of concurrent >> tasks to 512 (taskThrottle=512) and by specifying that maxJobs=32 and >> tasksPerWorker=16. If there are a total of 2430 total tasks to run, >> there will be 2430 - 2048 = 382 tasks left for the last ?go >> around? (assuming that each of the tasks take about the same time to >> complete). If there are 382 tasks left towards run completion and 512 >> cores (16 for each of the 32 nodes), would the last few jobs, end up >> spread across the 32 Nodes, effectively holding them and burning your >> allocation, or would they get pushed to 24 nodes and de-allocating 8 >> nodes? >> >> Jonathan >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > From hategan at mcs.anl.gov Tue Aug 12 13:55:54 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 12 Aug 2014 11:55:54 -0700 Subject: [Swift-user] Swift and allocations In-Reply-To: <0443760C-DE97-4DD9-859E-A3B62DC9584B@uchicago.edu> References: <1407865038.12289.3.camel@echo> <0443760C-DE97-4DD9-859E-A3B62DC9584B@uchicago.edu> Message-ID: <1407869754.12887.7.camel@echo> On Tue, 2014-08-12 at 13:45 -0500, Jonathan Ozik wrote: > Hi Mihael, > > Thank you, good to know. > The jobWalltime, should that be thought of as the length of time that > each block will stay up? > For example, if resources are tight and only a few blocks are > started, while the rest of the blocks are waiting for resources to > free up, if the first few blocks reach their jobWalltimes, will those > be shut down? jobWalltime in 0.95 isn't quite the correct term. It's the replacement for maxTime in pre-0.95 and it's a cap to the block walltime, which is otherwise calculated using the overallocation formula. But yes, whatever the block walltime, the block will be shut down after that. If maxTime/jobWalltime is specified, it is guaranteed that no block will have a walltime larger than maxTime/jobWalltime. I was probably lying a little before. There is some code to attempt to offload work to fewer blocks and shut down the unneeded ones, but I'm not entirely sure how well it works in your scenario. Mihael From Matthew.Shaxted at som.com Wed Aug 20 16:59:53 2014 From: Matthew.Shaxted at som.com (Matthew Shaxted) Date: Wed, 20 Aug 2014 17:59:53 -0400 Subject: [Swift-user] Dynamic For Loops from Input File Message-ID: Hi Swift Users, I have a swift script that reads an input file of parameters and variables, and iterates through the variables in one for loop per parameter: Input Sweep File Example: pnum pname pvals 1 InfilRate 0.0015,0.025 2 PreheatCoilSetpoint 15,15.5,16.0 3 KitchenGasPD 200,250 Swift File For Loops: foreach v0 in pval[0] { foreach v1 in pval[1] { foreach v2 in pval[2] { As of now, I need to hard code in the number of for loops in the Swift script to match those of the input file parameters, and I'm thinking there must be a better way. I'm wondering if there is a more dynamic way to have Swift loop through the parameters without having to hard code in the for loops? For example, if I define 5 parameters in the input file, the Swift script would automatically know to add 5 for loops. I was thinking I could create a shell script to read the input file and dynamically create the swift script with the correct number of for loops before running swift, but perhaps there is a better way? Thanks, Matthew MATTHEW SHAXTED SKIDMORE, OWINGS & MERRILL LLP 224 SOUTH MICHIGAN AVENUE CHICAGO, IL 60604 T (312) 360-4368 MATTHEW.SHAXTED at SOM.COM [cid:image004.png at 01CFBC98.297A1030] The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. [cid:image003.gif at 01CFBC96.E755A620] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.gif Type: image/gif Size: 566 bytes Desc: image003.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.png Type: image/png Size: 5311 bytes Desc: image004.png URL: From hategan at mcs.anl.gov Wed Aug 20 17:40:59 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 20 Aug 2014 15:40:59 -0700 Subject: [Swift-user] Dynamic For Loops from Input File In-Reply-To: References: Message-ID: <1408574459.15109.9.camel@echo> Hi Matthew, I'm thinking something like this: type var { string name; float[] values; } (float[] r) extend(float[] a, float v, int lastIndex) { foreach ov, i in a { r[i] = a[i]; } r[lastIndex] = v; } doSomething(float[] a) { trace(a); } dynamicFor(var[] vars, float[] values, int index, int len) { if (index < len) { tracef("Iterating over %s\n", vars[index].name); foreach v in vars[index].values { dynamicFor(vars, extend(values, v, index), index + 1, len); } } else { doSomething(values); } } var[] vars; vars[0].name = "a"; vars[0].values = [0.1, 0.2, 0.3]; vars[1].name = "b"; vars[1].values = [0.4, 0.5]; float[] empty; dynamicFor(vars, empty, 0, 2); Mihael On Wed, 2014-08-20 at 17:59 -0400, Matthew Shaxted wrote: > Hi Swift Users, > > I have a swift script that reads an input file of parameters and variables, and iterates through the variables in one for loop per parameter: > > Input Sweep File Example: > pnum pname pvals > 1 InfilRate 0.0015,0.025 > 2 PreheatCoilSetpoint 15,15.5,16.0 > 3 KitchenGasPD 200,250 > > Swift File For Loops: > foreach v0 in pval[0] { > foreach v1 in pval[1] { > foreach v2 in pval[2] { > > As of now, I need to hard code in the number of for loops in the Swift script to match those of the input file parameters, and I'm thinking there must be a better way. > > I'm wondering if there is a more dynamic way to have Swift loop through the parameters without having to hard code in the for loops? For example, if I define 5 parameters in the input file, the Swift script would automatically know to add 5 for loops. > > I was thinking I could create a shell script to read the input file and dynamically create the swift script with the correct number of for loops before running swift, but perhaps there is a better way? > > Thanks, > Matthew > > > MATTHEW SHAXTED > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > [cid:image004.png at 01CFBC98.297A1030] > The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. > > [cid:image003.gif at 01CFBC96.E755A620] > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From Matthew.Shaxted at som.com Wed Aug 20 17:51:09 2014 From: Matthew.Shaxted at som.com (Matthew Shaxted) Date: Wed, 20 Aug 2014 18:51:09 -0400 Subject: [Swift-user] Dynamic For Loops from Input File In-Reply-To: <1408574459.15109.9.camel@echo> References: <1408574459.15109.9.camel@echo> Message-ID: Hi Mihael, This looks like it may work well - many thanks for the quick response. I'll see if I can integrate this into my existing sweep scripts now. Thanks again, Matthew MATTHEW SHAXTED SKIDMORE, OWINGS & MERRILL LLP 224 SOUTH MICHIGAN AVENUE CHICAGO, IL 60604 T? (312) 360-4368 MATTHEW.SHAXTED at SOM.COM The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. -----Original Message----- From: Mihael Hategan [mailto:hategan at mcs.anl.gov] Sent: Wednesday, August 20, 2014 5:41 PM To: Matthew Shaxted Cc: swift-user at ci.uchicago.edu; Jason Kirkpatrick Subject: Re: [Swift-user] Dynamic For Loops from Input File Hi Matthew, I'm thinking something like this: type var { string name; float[] values; } (float[] r) extend(float[] a, float v, int lastIndex) { foreach ov, i in a { r[i] = a[i]; } r[lastIndex] = v; } doSomething(float[] a) { trace(a); } dynamicFor(var[] vars, float[] values, int index, int len) { if (index < len) { tracef("Iterating over %s\n", vars[index].name); foreach v in vars[index].values { dynamicFor(vars, extend(values, v, index), index + 1, len); } } else { doSomething(values); } } var[] vars; vars[0].name = "a"; vars[0].values = [0.1, 0.2, 0.3]; vars[1].name = "b"; vars[1].values = [0.4, 0.5]; float[] empty; dynamicFor(vars, empty, 0, 2); Mihael On Wed, 2014-08-20 at 17:59 -0400, Matthew Shaxted wrote: > Hi Swift Users, > > I have a swift script that reads an input file of parameters and variables, and iterates through the variables in one for loop per parameter: > > Input Sweep File Example: > pnum pname pvals > 1 InfilRate 0.0015,0.025 > 2 PreheatCoilSetpoint 15,15.5,16.0 > 3 KitchenGasPD 200,250 > > Swift File For Loops: > foreach v0 in pval[0] { > foreach v1 in pval[1] { > foreach v2 in pval[2] { > > As of now, I need to hard code in the number of for loops in the Swift script to match those of the input file parameters, and I'm thinking there must be a better way. > > I'm wondering if there is a more dynamic way to have Swift loop through the parameters without having to hard code in the for loops? For example, if I define 5 parameters in the input file, the Swift script would automatically know to add 5 for loops. > > I was thinking I could create a shell script to read the input file and dynamically create the swift script with the correct number of for loops before running swift, but perhaps there is a better way? > > Thanks, > Matthew > > > MATTHEW SHAXTED > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > [cid:image004.png at 01CFBC98.297A1030] > The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. > > [cid:image003.gif at 01CFBC96.E755A620] > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From Matthew.Shaxted at som.com Thu Aug 21 13:33:18 2014 From: Matthew.Shaxted at som.com (Matthew Shaxted) Date: Thu, 21 Aug 2014 14:33:18 -0400 Subject: [Swift-user] Dynamic For Loops from Input File References: <1408574459.15109.9.camel@echo> Message-ID: Thanks Mihael, I'm pleased to say this dynamic for loop code works well. Below is how my iterating script turned out. One thing I was not yet able to replicate is the counting of the total iterations in the Swift run. When I was previously hard coding the values, I was doing something similar to below - any chance you know of an easy to do this in the dynamic way? I was thinking I could iterate in the PARAMETER SETUP & LAUNCH section, but not quite sure how to best implement. int countRuns=@length(pval[0])*@length(pval[1])*@length(pval[2])*@length(pval[3])*@length(pval[4])*@length(pval[5])*@length(pval[6])*@length(pval[7])*@length(pval[8]); tracef("%i Runs in Simulation\n\n",countRuns); Thanks, Matthew # ----- ENERGYPLUS SWEEP SWIFT SCRIPT ----- # # ----- TYPE DEFINITIONS type file; type param { string pname; string pvals; } type pval { string name; string[] values; } # ----- APP DEFINITIONS app (file all) runEP ( file imf, file epw, string params[] ) { RunEPlocal "--imf" @imf "--epw" @epw "--outall" @all "--params" params; } # ----- EXTERNAL INPUTS file epconfig ; file epweather ; string outdir=@arg("outdir","output"); string count=@arg("count","false"); param pset[] = readData(@arg("sweep","test.sweep")); # ----- INTERNAL FUNCTIONS (string[] r) extend(string[] a, string v,string n, int lastIndex) { foreach ov, i in a {r[i] = a[i];} string[] f;f[0]=n;f[1]=v; r[lastIndex]=@strjoin(f,","); } iterateEP(file c, file w, string[] parameters, string d) { string ap=@strjoin(parameters,","); string[] apx=@strsplit(ap,","); string[] aval; foreach v, vn in parameters { string[] an=@strsplit(v,","); aval[vn]=an[1]; } string fileid = @strcat("ep+", at strjoin(aval,"+")); tracef("%s\n",fileid); file outall ; (outall) = runEP (c,w,apx); } dynamicFor(pval[] vars, string[] values, int index, int len, file epc, file epw, string od) { if (index < len) { foreach v in vars[index].values {dynamicFor(vars, extend(values, v,vars[index].name, index), index + 1, len, epc, epw, od);} }else {iterateEP(epc,epw,values,od);} } # ----- PARAMETER SETUP & LAUNCH string[] empty; pval[] pvals; foreach p, pn in pset { string val[]=@strsplit(p.pvals,","); pvals[pn].name=p.pname; foreach v, vn in val { pvals[pn].values[vn]=v; } } dynamicFor(pvals, empty, 0, @length(pvals), epconfig, epweather, outdir); -----Original Message----- From: Matthew Shaxted Sent: Wednesday, August 20, 2014 5:51 PM To: 'Mihael Hategan' Cc: swift-user at ci.uchicago.edu; Jason Kirkpatrick Subject: RE: [Swift-user] Dynamic For Loops from Input File Hi Mihael, This looks like it may work well - many thanks for the quick response. I'll see if I can integrate this into my existing sweep scripts now. Thanks again, Matthew MATTHEW SHAXTED SKIDMORE, OWINGS & MERRILL LLP 224 SOUTH MICHIGAN AVENUE CHICAGO, IL 60604 T? (312) 360-4368 MATTHEW.SHAXTED at SOM.COM The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. -----Original Message----- From: Mihael Hategan [mailto:hategan at mcs.anl.gov] Sent: Wednesday, August 20, 2014 5:41 PM To: Matthew Shaxted Cc: swift-user at ci.uchicago.edu; Jason Kirkpatrick Subject: Re: [Swift-user] Dynamic For Loops from Input File Hi Matthew, I'm thinking something like this: type var { string name; float[] values; } (float[] r) extend(float[] a, float v, int lastIndex) { foreach ov, i in a { r[i] = a[i]; } r[lastIndex] = v; } doSomething(float[] a) { trace(a); } dynamicFor(var[] vars, float[] values, int index, int len) { if (index < len) { tracef("Iterating over %s\n", vars[index].name); foreach v in vars[index].values { dynamicFor(vars, extend(values, v, index), index + 1, len); } } else { doSomething(values); } } var[] vars; vars[0].name = "a"; vars[0].values = [0.1, 0.2, 0.3]; vars[1].name = "b"; vars[1].values = [0.4, 0.5]; float[] empty; dynamicFor(vars, empty, 0, 2); Mihael On Wed, 2014-08-20 at 17:59 -0400, Matthew Shaxted wrote: > Hi Swift Users, > > I have a swift script that reads an input file of parameters and variables, and iterates through the variables in one for loop per parameter: > > Input Sweep File Example: > pnum pname pvals > 1 InfilRate 0.0015,0.025 > 2 PreheatCoilSetpoint 15,15.5,16.0 > 3 KitchenGasPD 200,250 > > Swift File For Loops: > foreach v0 in pval[0] { > foreach v1 in pval[1] { > foreach v2 in pval[2] { > > As of now, I need to hard code in the number of for loops in the Swift script to match those of the input file parameters, and I'm thinking there must be a better way. > > I'm wondering if there is a more dynamic way to have Swift loop through the parameters without having to hard code in the for loops? For example, if I define 5 parameters in the input file, the Swift script would automatically know to add 5 for loops. > > I was thinking I could create a shell script to read the input file and dynamically create the swift script with the correct number of for loops before running swift, but perhaps there is a better way? > > Thanks, > Matthew > > > MATTHEW SHAXTED > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > [cid:image004.png at 01CFBC98.297A1030] > The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. > > [cid:image003.gif at 01CFBC96.E755A620] > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From hategan at mcs.anl.gov Thu Aug 21 13:53:00 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 21 Aug 2014 11:53:00 -0700 Subject: [Swift-user] Dynamic For Loops from Input File In-Reply-To: References: <1408574459.15109.9.camel@echo> Message-ID: <1408647180.1970.7.camel@echo> Hi, Same idea I would think: (int sz) countAllRecursive(sometype[] pval, int index) { if (index < 0) { sz = 1; } else { sz = length(pval[index]) * countAllRecursive(pval, index - 1); } } (int sz) countAll(sometype[] pval) { if (length(pval) == 0) { sz = 0; } else { sz = countAllRecursive(pval, length(pval) - 1); } } Mihael On Thu, 2014-08-21 at 14:33 -0400, Matthew Shaxted wrote: > Thanks Mihael, > > I'm pleased to say this dynamic for loop code works well. Below is how my iterating script turned out. > > One thing I was not yet able to replicate is the counting of the total iterations in the Swift run. > > When I was previously hard coding the values, I was doing something similar to below - any chance you know of an easy to do this in the dynamic way? I was thinking I could iterate in the PARAMETER SETUP & LAUNCH section, but not quite sure how to best implement. > > int countRuns=@length(pval[0])*@length(pval[1])*@length(pval[2])*@length(pval[3])*@length(pval[4])*@length(pval[5])*@length(pval[6])*@length(pval[7])*@length(pval[8]); > tracef("%i Runs in Simulation\n\n",countRuns); > > Thanks, > Matthew > > > > # ----- ENERGYPLUS SWEEP SWIFT SCRIPT ----- # > > # ----- TYPE DEFINITIONS > > type file; > > type param { > string pname; > string pvals; > } > > type pval { > string name; > string[] values; > } > > > # ----- APP DEFINITIONS > > app (file all) runEP ( file imf, file epw, string params[] ) > { > RunEPlocal "--imf" @imf "--epw" @epw "--outall" @all "--params" params; > } > > > # ----- EXTERNAL INPUTS > > file epconfig ; > file epweather ; > string outdir=@arg("outdir","output"); > string count=@arg("count","false"); > param pset[] = readData(@arg("sweep","test.sweep")); > > > # ----- INTERNAL FUNCTIONS > > (string[] r) extend(string[] a, string v,string n, int lastIndex) { > foreach ov, i in a {r[i] = a[i];} > string[] f;f[0]=n;f[1]=v; > r[lastIndex]=@strjoin(f,","); > } > > iterateEP(file c, file w, string[] parameters, string d) { > string ap=@strjoin(parameters,","); > string[] apx=@strsplit(ap,","); > string[] aval; > foreach v, vn in parameters { > string[] an=@strsplit(v,","); > aval[vn]=an[1]; > } > string fileid = @strcat("ep+", at strjoin(aval,"+")); > tracef("%s\n",fileid); > file outall ; > (outall) = runEP (c,w,apx); > } > > dynamicFor(pval[] vars, string[] values, int index, int len, file epc, file epw, string od) { > if (index < len) { > foreach v in vars[index].values {dynamicFor(vars, extend(values, v,vars[index].name, index), index + 1, len, epc, epw, od);} > }else {iterateEP(epc,epw,values,od);} > } > > > # ----- PARAMETER SETUP & LAUNCH > > string[] empty; > pval[] pvals; > > foreach p, pn in pset { > string val[]=@strsplit(p.pvals,","); > pvals[pn].name=p.pname; > foreach v, vn in val { > pvals[pn].values[vn]=v; > } > } > > dynamicFor(pvals, empty, 0, @length(pvals), epconfig, epweather, outdir); > > > > > > > -----Original Message----- > From: Matthew Shaxted > Sent: Wednesday, August 20, 2014 5:51 PM > To: 'Mihael Hategan' > Cc: swift-user at ci.uchicago.edu; Jason Kirkpatrick > Subject: RE: [Swift-user] Dynamic For Loops from Input File > > Hi Mihael, > > This looks like it may work well - many thanks for the quick response. > > I'll see if I can integrate this into my existing sweep scripts now. > > Thanks again, > Matthew > > > MATTHEW SHAXTED > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > > The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. > > > > -----Original Message----- > From: Mihael Hategan [mailto:hategan at mcs.anl.gov] > Sent: Wednesday, August 20, 2014 5:41 PM > To: Matthew Shaxted > Cc: swift-user at ci.uchicago.edu; Jason Kirkpatrick > Subject: Re: [Swift-user] Dynamic For Loops from Input File > > Hi Matthew, > > I'm thinking something like this: > > type var { > string name; > float[] values; > } > > (float[] r) extend(float[] a, float v, int lastIndex) { > foreach ov, i in a { > r[i] = a[i]; > } > r[lastIndex] = v; > } > > doSomething(float[] a) { > trace(a); > } > > dynamicFor(var[] vars, float[] values, int index, int len) { > if (index < len) { > tracef("Iterating over %s\n", vars[index].name); > foreach v in vars[index].values { > dynamicFor(vars, extend(values, v, index), index + 1, len); > } > } > else { > doSomething(values); > } > } > > var[] vars; > vars[0].name = "a"; > vars[0].values = [0.1, 0.2, 0.3]; > vars[1].name = "b"; > vars[1].values = [0.4, 0.5]; > > float[] empty; > > dynamicFor(vars, empty, 0, 2); > > Mihael > > > On Wed, 2014-08-20 at 17:59 -0400, Matthew Shaxted wrote: > > Hi Swift Users, > > > > I have a swift script that reads an input file of parameters and variables, and iterates through the variables in one for loop per parameter: > > > > Input Sweep File Example: > > pnum pname pvals > > 1 InfilRate 0.0015,0.025 > > 2 PreheatCoilSetpoint 15,15.5,16.0 > > 3 KitchenGasPD 200,250 > > > > Swift File For Loops: > > foreach v0 in pval[0] { > > foreach v1 in pval[1] { > > foreach v2 in pval[2] { > > > > As of now, I need to hard code in the number of for loops in the Swift script to match those of the input file parameters, and I'm thinking there must be a better way. > > > > I'm wondering if there is a more dynamic way to have Swift loop through the parameters without having to hard code in the for loops? For example, if I define 5 parameters in the input file, the Swift script would automatically know to add 5 for loops. > > > > I was thinking I could create a shell script to read the input file and dynamically create the swift script with the correct number of for loops before running swift, but perhaps there is a better way? > > > > Thanks, > > Matthew > > > > > > MATTHEW SHAXTED > > SKIDMORE, OWINGS & MERRILL LLP > > 224 SOUTH MICHIGAN AVENUE > > CHICAGO, IL 60604 > > T (312) 360-4368 > > MATTHEW.SHAXTED at SOM.COM > > > > [cid:image004.png at 01CFBC98.297A1030] > > The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. > > > > [cid:image003.gif at 01CFBC96.E755A620] > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > From Matthew.Shaxted at som.com Thu Aug 21 14:18:42 2014 From: Matthew.Shaxted at som.com (Matthew Shaxted) Date: Thu, 21 Aug 2014 15:18:42 -0400 Subject: [Swift-user] Dynamic For Loops from Input File In-Reply-To: <1408647180.1970.7.camel@echo> References: <1408574459.15109.9.camel@echo> <1408647180.1970.7.camel@echo> Message-ID: Ah yes Mihael this is it... attached my finished swift script for reference. Thanks again for your help. # ----- TYPE DEFINITIONS type file; type param { string pname; string pvals; } type pval { string name; string[] values; } # ----- APP DEFINITIONS app (file all) runEP ( file imf, file epw, string params[] ) { RunEPlocal "--imf" @imf "--epw" @epw "--outall" @all "--params" params; } # ----- EXTERNAL INPUTS file epconfig ; file epweather ; string outdir=@arg("outdir","output"); string count=@arg("count","true"); param pset[] = readData(@arg("sweep","test.sweep")); # ----- INTERNAL FUNCTIONS (string[] r) extend(string[] a, string v,string n, int lastIndex) { foreach ov, i in a {r[i] = a[i];} string[] f;f[0]=n;f[1]=v; r[lastIndex]=@strjoin(f,","); } iterateEP(file c, file w, string[] parameters, string d) { string ap=@strjoin(parameters,","); string[] apx=@strsplit(ap,","); string[] aval; foreach v, vn in parameters { string[] an=@strsplit(v,","); aval[vn]=an[1]; } string fileid = @strcat("ep+", at strjoin(aval,"+")); tracef("%s\n",fileid); file outall ; (outall) = runEP (c,w,apx); } dynamicFor(pval[] vars, string[] values, int index, int len, file epc, file epw, string od) { if (index < len) { foreach v in vars[index].values {dynamicFor(vars, extend(values, v,vars[index].name, index), index + 1, len, epc, epw, od);} }else {iterateEP(epc,epw,values,od);} } (int sz) countAllRecursive(pval[] pval, int index) { if (index < 0) {sz = 1;} else {sz = @length(pval[index].values) * countAllRecursive(pval, index - 1);} } (int sz) countAll(pval[] pval) { if (@length(pval) == 0) {sz = 0;} else {sz = countAllRecursive(pval, @length(pval) - 1);} } # ----- PARAMETER SETUP & LAUNCH string[] empty; pval[] pvals; foreach p, pn in pset { string val[]=@strsplit(p.pvals,","); pvals[pn].name=p.pname; foreach v, vn in val { pvals[pn].values[vn]=v; } } int runs=countAll(pvals); tracef("%i Runs in Simulation\n\n",runs); if (count=="false"){ dynamicFor(pvals, empty, 0, @length(pvals), epconfig, epweather, outdir); } -----Original Message----- From: Mihael Hategan [mailto:hategan at mcs.anl.gov] Sent: Thursday, August 21, 2014 1:53 PM To: Matthew Shaxted Cc: swift-user at ci.uchicago.edu; Jason Kirkpatrick Subject: Re: [Swift-user] Dynamic For Loops from Input File Hi, Same idea I would think: (int sz) countAllRecursive(sometype[] pval, int index) { if (index < 0) { sz = 1; } else { sz = length(pval[index]) * countAllRecursive(pval, index - 1); } } (int sz) countAll(sometype[] pval) { if (length(pval) == 0) { sz = 0; } else { sz = countAllRecursive(pval, length(pval) - 1); } } Mihael On Thu, 2014-08-21 at 14:33 -0400, Matthew Shaxted wrote: > Thanks Mihael, > > I'm pleased to say this dynamic for loop code works well. Below is how my iterating script turned out. > > One thing I was not yet able to replicate is the counting of the total iterations in the Swift run. > > When I was previously hard coding the values, I was doing something similar to below - any chance you know of an easy to do this in the dynamic way? I was thinking I could iterate in the PARAMETER SETUP & LAUNCH section, but not quite sure how to best implement. > > int countRuns=@length(pval[0])*@length(pval[1])*@length(pval[2])*@length(pval[3])*@length(pval[4])*@length(pval[5])*@length(pval[6])*@length(pval[7])*@length(pval[8]); > tracef("%i Runs in Simulation\n\n",countRuns); > > Thanks, > Matthew > > > > # ----- ENERGYPLUS SWEEP SWIFT SCRIPT ----- # > > # ----- TYPE DEFINITIONS > > type file; > > type param { > string pname; > string pvals; > } > > type pval { > string name; > string[] values; > } > > > # ----- APP DEFINITIONS > > app (file all) runEP ( file imf, file epw, string params[] ) > { > RunEPlocal "--imf" @imf "--epw" @epw "--outall" @all "--params" params; > } > > > # ----- EXTERNAL INPUTS > > file epconfig ; > file epweather ; > string outdir=@arg("outdir","output"); > string count=@arg("count","false"); > param pset[] = readData(@arg("sweep","test.sweep")); > > > # ----- INTERNAL FUNCTIONS > > (string[] r) extend(string[] a, string v,string n, int lastIndex) { > foreach ov, i in a {r[i] = a[i];} > string[] f;f[0]=n;f[1]=v; > r[lastIndex]=@strjoin(f,","); > } > > iterateEP(file c, file w, string[] parameters, string d) { > string ap=@strjoin(parameters,","); > string[] apx=@strsplit(ap,","); > string[] aval; > foreach v, vn in parameters { > string[] an=@strsplit(v,","); > aval[vn]=an[1]; > } > string fileid = @strcat("ep+", at strjoin(aval,"+")); > tracef("%s\n",fileid); > file outall ; > (outall) = runEP (c,w,apx); > } > > dynamicFor(pval[] vars, string[] values, int index, int len, file epc, file epw, string od) { > if (index < len) { > foreach v in vars[index].values {dynamicFor(vars, extend(values, v,vars[index].name, index), index + 1, len, epc, epw, od);} > }else {iterateEP(epc,epw,values,od);} > } > > > # ----- PARAMETER SETUP & LAUNCH > > string[] empty; > pval[] pvals; > > foreach p, pn in pset { > string val[]=@strsplit(p.pvals,","); > pvals[pn].name=p.pname; > foreach v, vn in val { > pvals[pn].values[vn]=v; > } > } > > dynamicFor(pvals, empty, 0, @length(pvals), epconfig, epweather, outdir); > > > > > > > -----Original Message----- > From: Matthew Shaxted > Sent: Wednesday, August 20, 2014 5:51 PM > To: 'Mihael Hategan' > Cc: swift-user at ci.uchicago.edu; Jason Kirkpatrick > Subject: RE: [Swift-user] Dynamic For Loops from Input File > > Hi Mihael, > > This looks like it may work well - many thanks for the quick response. > > I'll see if I can integrate this into my existing sweep scripts now. > > Thanks again, > Matthew > > > MATTHEW SHAXTED > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > > The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. > > > > -----Original Message----- > From: Mihael Hategan [mailto:hategan at mcs.anl.gov] > Sent: Wednesday, August 20, 2014 5:41 PM > To: Matthew Shaxted > Cc: swift-user at ci.uchicago.edu; Jason Kirkpatrick > Subject: Re: [Swift-user] Dynamic For Loops from Input File > > Hi Matthew, > > I'm thinking something like this: > > type var { > string name; > float[] values; > } > > (float[] r) extend(float[] a, float v, int lastIndex) { > foreach ov, i in a { > r[i] = a[i]; > } > r[lastIndex] = v; > } > > doSomething(float[] a) { > trace(a); > } > > dynamicFor(var[] vars, float[] values, int index, int len) { > if (index < len) { > tracef("Iterating over %s\n", vars[index].name); > foreach v in vars[index].values { > dynamicFor(vars, extend(values, v, index), index + 1, len); > } > } > else { > doSomething(values); > } > } > > var[] vars; > vars[0].name = "a"; > vars[0].values = [0.1, 0.2, 0.3]; > vars[1].name = "b"; > vars[1].values = [0.4, 0.5]; > > float[] empty; > > dynamicFor(vars, empty, 0, 2); > > Mihael > > > On Wed, 2014-08-20 at 17:59 -0400, Matthew Shaxted wrote: > > Hi Swift Users, > > > > I have a swift script that reads an input file of parameters and variables, and iterates through the variables in one for loop per parameter: > > > > Input Sweep File Example: > > pnum pname pvals > > 1 InfilRate 0.0015,0.025 > > 2 PreheatCoilSetpoint 15,15.5,16.0 > > 3 KitchenGasPD 200,250 > > > > Swift File For Loops: > > foreach v0 in pval[0] { > > foreach v1 in pval[1] { > > foreach v2 in pval[2] { > > > > As of now, I need to hard code in the number of for loops in the Swift script to match those of the input file parameters, and I'm thinking there must be a better way. > > > > I'm wondering if there is a more dynamic way to have Swift loop through the parameters without having to hard code in the for loops? For example, if I define 5 parameters in the input file, the Swift script would automatically know to add 5 for loops. > > > > I was thinking I could create a shell script to read the input file and dynamically create the swift script with the correct number of for loops before running swift, but perhaps there is a better way? > > > > Thanks, > > Matthew > > > > > > MATTHEW SHAXTED > > SKIDMORE, OWINGS & MERRILL LLP > > 224 SOUTH MICHIGAN AVENUE > > CHICAGO, IL 60604 > > T (312) 360-4368 > > MATTHEW.SHAXTED at SOM.COM > > > > [cid:image004.png at 01CFBC98.297A1030] > > The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. > > > > [cid:image003.gif at 01CFBC96.E755A620] > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > From ianbriggs at comcast.net Wed Aug 20 17:19:57 2014 From: ianbriggs at comcast.net (Ian Briggs) Date: Wed, 20 Aug 2014 16:19:57 -0600 Subject: [Swift-user] Swift/K node creation Message-ID: <804C610C-90B8-4E2B-96A3-4186D5E37877@comcast.net> Thank you for reading this. I was able to install and run swift/k on my local computer. From here I would like to install it on a remote machine and use that machine as a remote site. How should I go about doing this? If there is a guide I missed I apologize. Thanks, Ian Briggs From yadunand at uchicago.edu Fri Aug 22 13:39:58 2014 From: yadunand at uchicago.edu (Yadu Nand Babuji) Date: Fri, 22 Aug 2014 13:39:58 -0500 Subject: [Swift-user] Fwd: Fwd: Swift/K node creation In-Reply-To: <53F6295F.5040402@anl.gov> References: <53F51FF7.7010107@cs.utah.edu> <53F6295F.5040402@anl.gov> Message-ID: <53F78E7E.9010704@uchicago.edu> Hi Ian, Reposting as I suspect that the original email bounced. Here's a tar ball of the swift tutorial with minor changes to the swift.properties config file to connect to a few remote machines : http://users.rcc.uchicago.edu/~yadunand/remote_pool.tar.gz Before you run the tests, please make sure to edit the remote_pool/swift.properties file to have the URLs of your remote machines. There's an online swift sandbox here -> http://swift-lang.org/tryswift/ You can get documentation here for different releases of Swift -> http://swift-lang.org/docs/index.php I would recommend getting Swift 0.95-RC6 from (http://swift-lang.org/downloads/index.php). Thanks! Yadu On 08/21/2014 12:16 PM, Michael Wilde wrote: > > > > -------- Original Message -------- > Subject: Fwd: Swift/K node creation > Date: Wed, 20 Aug 2014 16:23:51 -0600 > From: Ganesh Gopalakrishnan > To: Michael Wilde > CC: Martin Burtscher , ian briggs > , Mark Baranowski > > > > Hi Mike, > > After yesterday's call, Ian Briggs and Mark Baranowski have looked > into Swift/K. I encouraged them to post a question to the Swift > group. Hence this email is strictly redundant - but I thought I'd > drop in an intro to give you some context for this question. Thanks! > > Best, > > Ganesh > > > -------- Original Message -------- > Subject: Swift/K node creation > Date: Wed, 20 Aug 2014 16:19:57 -0600 > From: Ian Briggs > To: swift-user at ci.uchicago.edu > CC: Ganesh Gopalakrishnan > > > > Thank you for reading this. > > I was able to install and run swift/k on my local computer. > From here I would like to install it on a remote machine > and use that machine as a remote site. How should I go > about doing this? If there is a guide I missed I apologize. > > Thanks, > Ian Briggs > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wozniak at mcs.anl.gov Fri Aug 22 08:30:23 2014 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Fri, 22 Aug 2014 15:30:23 +0200 Subject: [Swift-user] Queuing isssue with SWFIT on Orthros, APS cluster In-Reply-To: <53A9B631.70706@anl.gov> References: <53A9B631.70706@anl.gov> Message-ID: <53F745EF.9040807@mcs.anl.gov> This message was held in our mailing list- is this still an issue? On 6/24/2014 7:32 PM, Hemant Sharma wrote: > Hi guys, > > I'm having a problem with queuing using SWIFT. I have a swift script, > which should execute about 27000 iterations. In order to limit the > initial memory size, I created the following config file: > > use.provider.staging=false > provider.staging.pin.swiftfiles=false > use.wrapper.staging=false > status.mode=provider > wrapperlog.always.transfer=true > execution.retries=0 > lazy.errors=true > sitedir.keep=true > file.gc.enabled=false > wrapper.parameter.mode=files > foreach.max.threads=330 > > Some times, when I execute the script, it starts with 320 active jobs > (on 320 processors), but after some time, it just gets stuck with 330 > submitted jobs and none of them are active. Example output to screen is: > > Swift 0.94 swift-r6637 cog-r3742 > > RunID: 20140624-1200-qke4z0v8 > Progress: time: Tue, 24 Jun 2014 12:00:30 -0500 > Progress: time: Tue, 24 Jun 2014 12:00:31 -0500 Selecting site:328 > Initializing site shared directory:1 Stage in:1 > Progress: time: Tue, 24 Jun 2014 12:00:32 -0500 Selecting site:10 > Stage in:277 Submitting:3 Submitted:40 > Progress: time: Tue, 24 Jun 2014 12:00:38 -0500 Selecting site:10 > Submitted:319 Active:1 > Progress: time: Tue, 24 Jun 2014 12:00:43 -0500 Selecting site:10 > Active:319 Checking status:1 > Progress: time: Tue, 24 Jun 2014 12:00:44 -0500 Selecting site:1 > Stage in:20 Active:208 Checking status:31 Stage out:70 Finished > successfully:21 > Progress: time: Tue, 24 Jun 2014 12:00:45 -0500 Stage in:11 > Active:120 Checking status:10 Stage out:189 Finished successfully:31 > Progress: time: Tue, 24 Jun 2014 12:00:46 -0500 Stage in:25 > Active:117 Stage out:188 Finished successfully:54 > Progress: time: Tue, 24 Jun 2014 12:00:47 -0500 Initializing:1 > Selecting site:1 Stage in:46 Active:118 Stage out:164 Finished > successfully:86 > Progress: time: Tue, 24 Jun 2014 12:00:48 -0500 Selecting site:2 > Stage in:102 Submitting:1 Submitted:2 Active:165 Checking > status:1 Stage out:57 Finished successfully:199 > Progress: time: Tue, 24 Jun 2014 12:00:49 -0500 Submitted:5 > Active:324 Checking status:1 Finished successfully:265 > Progress: time: Tue, 24 Jun 2014 12:00:50 -0500 Submitted:12 > Active:317 Checking status:1 Finished successfully:272 > Progress: time: Tue, 24 Jun 2014 12:00:51 -0500 Submitted:22 > Active:307 Finished successfully:283 > Progress: time: Tue, 24 Jun 2014 12:00:52 -0500 Selecting site:1 > Stage in:13 Submitted:47 Active:223 Stage out:46 Finished > successfully:321 > Progress: time: Tue, 24 Jun 2014 12:00:53 -0500 Stage in:28 > Submitted:73 Active:153 Stage out:75 Finished successfully:362 > Progress: time: Tue, 24 Jun 2014 12:00:55 -0500 Submitted:182 > Active:147 Checking status:1 Finished successfully:442 > Progress: time: Tue, 24 Jun 2014 12:00:57 -0500 Submitted:183 > Active:146 Checking status:1 Finished successfully:443 > Progress: time: Tue, 24 Jun 2014 12:01:00 -0500 Submitted:185 > Active:144 Checking status:1 Finished successfully:445 > Progress: time: Tue, 24 Jun 2014 12:01:01 -0500 Submitted:186 > Active:143 Checking status:1 Finished successfully:446 > Progress: time: Tue, 24 Jun 2014 12:01:02 -0500 Submitted:190 > Active:139 Checking status:1 Finished successfully:450 > Progress: time: Tue, 24 Jun 2014 12:01:05 -0500 Submitted:193 > Active:136 Checking status:1 Finished successfully:453 > Progress: time: Tue, 24 Jun 2014 12:01:07 -0500 Submitted:196 > Active:133 Checking status:1 Finished successfully:456 > Progress: time: Tue, 24 Jun 2014 12:01:09 -0500 Submitted:198 > Active:131 Checking status:1 Finished successfully:458 > Progress: time: Tue, 24 Jun 2014 12:01:10 -0500 Stage in:5 > Submitted:202 Active:63 Stage out:60 Finished successfully:467 > Progress: time: Tue, 24 Jun 2014 12:01:11 -0500 Submitted:273 > Active:56 Checking status:1 Finished successfully:533 > Progress: time: Tue, 24 Jun 2014 12:01:13 -0500 Submitted:282 > Active:47 Checking status:1 Finished successfully:542 > Progress: time: Tue, 24 Jun 2014 12:01:14 -0500 Submitting:1 > Submitted:292 Active:37 Finished successfully:553 > Progress: time: Tue, 24 Jun 2014 12:01:15 -0500 Submitted:298 > Active:31 Checking status:1 Finished successfully:558 > Progress: time: Tue, 24 Jun 2014 12:01:16 -0500 Submitted:305 > Active:24 Checking status:1 Finished successfully:565 > Progress: time: Tue, 24 Jun 2014 12:01:17 -0500 Submitted:307 > Active:22 Checking status:1 Finished successfully:567 > Progress: time: Tue, 24 Jun 2014 12:01:18 -0500 Submitted:313 > Active:16 Checking status:1 Finished successfully:573 > Progress: time: Tue, 24 Jun 2014 12:01:20 -0500 Submitted:315 > Active:14 Checking status:1 Finished successfully:575 > Progress: time: Tue, 24 Jun 2014 12:01:21 -0500 Submitted:317 > Active:12 Checking status:1 Finished successfully:577 > Progress: time: Tue, 24 Jun 2014 12:01:22 -0500 Submitted:319 > Active:10 Checking status:1 Finished successfully:579 > Progress: time: Tue, 24 Jun 2014 12:01:23 -0500 Submitted:320 > Active:9 Checking status:1 Finished successfully:580 > Progress: time: Tue, 24 Jun 2014 12:01:25 -0500 Submitted:323 > Active:6 Checking status:1 Finished successfully:583 > Progress: time: Tue, 24 Jun 2014 12:01:26 -0500 Submitted:324 > Active:5 Checking status:1 Finished successfully:584 > Progress: time: Tue, 24 Jun 2014 12:01:27 -0500 Submitted:325 > Active:4 Checking status:1 Finished successfully:585 > Progress: time: Tue, 24 Jun 2014 12:01:29 -0500 Submitted:326 > Active:3 Checking status:1 Finished successfully:586 > Progress: time: Tue, 24 Jun 2014 12:01:36 -0500 Submitted:327 > Active:2 Checking status:1 Finished successfully:587 > Progress: time: Tue, 24 Jun 2014 12:01:39 -0500 Submitted:328 > Active:1 Checking status:1 Finished successfully:588 > Progress: time: Tue, 24 Jun 2014 12:01:50 -0500 Submitted:329 > Checking status:1 Finished successfully:589 > Progress: time: Tue, 24 Jun 2014 12:02:00 -0500 Submitted:330 > Finished successfully:590 > Progress: time: Tue, 24 Jun 2014 12:02:30 -0500 Submitted:330 > Finished successfully:590 > Progress: time: Tue, 24 Jun 2014 12:03:00 -0500 Submitted:330 > Finished successfully:590 > Progress: time: Tue, 24 Jun 2014 12:03:30 -0500 Submitted:330 > Finished successfully:590 > Progress: time: Tue, 24 Jun 2014 12:04:00 -0500 Submitted:330 > Finished successfully:590 > > The issue is not really reproducible, nor is the number of successful > jobs. Any ideas how to solve this problem? I'm attaching the log file. > > Thanks, > Hemant > > Hemant Sharma > Post-doctoral Researcher > Advanced Photon Source > Argonne National Laboratory > Lemont IL 60429 > USA > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Justin M Wozniak -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Sat Aug 23 15:09:50 2014 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sat, 23 Aug 2014 15:09:50 -0500 Subject: [Swift-user] CFP: ACM MTAGS 2014 @ SC14 -- deadline extension to September 8th & keynote speakers announced Message-ID: <53F8F50E.8080200@cs.iit.edu> Call for Papers --------------------------------------------------------------------------------------- The 7th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) 2014 http://datasys.cs.iit.edu/events/MTAGS14/ --------------------------------------------------------------------------------------- November 16th, 2014 New Orleans, Louisiana, USA Co-located with with IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC14) In cooperation with ACM SIGHPC NEWS ======================================================================================= - Deadline extension to September 8th, 2014 - Confirmed keynote speakers: - Dr. Owen O'Malley, Co-founder Hortonworks Inc. - Mike Wilde, Fellow at University of Chicago and Software Architect at Argonne National Laboratory - Call for Papers: Special Issue on Many-Task Computing in the Cloud, in the IEEE Transaction on Cloud Computing -- papers due February 2015 ======================================================================================= The 7th workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) will provide the scientific community a dedicated forum for presenting new research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications, which are generally composed of many tasks (both independent and dependent tasks) to achieve some larger application goal. This workshop will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions on all theoretical, simulations, and systems topics related to MTC, but we give special consideration to papers addressing petascale to exascale challenges. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the ACM digital library (pending approval). The workshop will be co-located with the IEEE/ACM Supercomputing 2014 Conference in New Orleans on November 17th, 2014. For more information, please see http://datasys.cs.iit.edu/events/MTAGS14/. For more information on past workshops, please see MTAGS13, MTAGS12, MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. We, the workshop organizers, also published a highly relevant paper that defines Many-Task Computing which was published in MTAGS08, titled ?Many-Task Computing for Grids and Supercomputers?; we encourage potential authors to read this paper, and to clearly articulate in your paper submissions how your papers are related to Many-Task Computing. Topics --------------------------------------------------------------------------------------- We invite the submission of original work that is related to the topics below. The papers should be 6 pages, including all figures and references. We aim to cover topics related to Many-Task Computing on each of the three major distributed systems paradigms, Cloud Computing, Grid Computing and Supercomputing. Topics of interest include: * Compute Resource Management * Scheduling * Job execution frameworks * Local resource manager extensions * Performance evaluation of resource managers in use on large scale systems * Dynamic resource provisioning * Techniques to manage many-core resources and/or GPUs * Challenges and opportunities in running many-task workloads on HPC systems * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure * Storage architectures and implementations * Distributed file systems * Parallel file systems * Distributed meta-data management * Content distribution systems for large data * Data caching frameworks and techniques * Data management within and across data centers * Data-aware scheduling * Data-intensive computing applications * Eventual-consistency storage usage and management * Programming models and tools * Map-reduce and its generalizations * Many-task computing middleware and applications * Parallel programming frameworks * Ensemble MPI techniques and frameworks * Service-oriented science applications * Large-Scale Workflow Systems * Workflow system performance and scalability analysis * Scalability of workflow systems * Workflow infrastructure and e-Science middleware * Programming Paradigms and Models * Large-Scale Many-Task Applications * High-throughput computing (HTC) applications * Data-intensive applications * Quasi-supercomputing applications, deployments, and experiences * Performance Evaluation * Performance evaluation * Real systems * Simulations * Reliability of large systems Paper Submission and Publication --------------------------------------------------------------------------------------- Authors are invited to submit papers with unpublished, original work of not more than 6 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages, as per ACM 8.5 x 11 manuscript guidelines; document templates can be found at http://www.acm.org/sigs/publications/proceedings-templates. The final 6 page papers in PDF format must be submitted online at https://cmt.research.microsoft.com/MTAGS2014/ before the deadline of September 8th, 2014 at 11:59PM PST. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the ACM digital library (in cooperation with SIGHPC). Notifications of the paper decisions will be sent out by September 22nd, 2014. Accepted workshop papers will be eligible for additional post-conference publication as journal articles in the IEEE Transaction on Cloud Computing, Special Issue on Many-Task Computing in the Cloud (papers will be due in February 2015, http://datasys.cs.iit.edu/events/TCC-MTC15/index.html). Submission implies the willingness of at least one of the authors to register and present the paper. For more information, please see http://datasys.cs.iit.edu/events/MTAGS14/. Important Dates --------------------------------------------------------------------------------------- * Papers Due: September 8th, 2014 * Notification of Acceptance: September 29th, 2014 * Camera Ready Papers Due: October 6th, 2014 * Workshop Date: November 16th, 2014 Committee Members --------------------------------------------------------------------------------------- Workshop Chairs * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory * Justin Wozniak, University of Chicago & Argonne National Laboratory * Ian Foster, University of Chicago & Argonne National Laboratory * Yong Zhao, University of Electronic Science and Technology of China Steering Committee * David Abramson, Monash University, Australia * Jack Dongarra, University of Tennessee, USA * Geoffrey Fox, Indiana University, USA * Manish Parashar, Rutgers University, USA * Marc Snir, University of Illinois at Urbana Champaign, USA * Xian-He Sun, Illinois Institute of Technology, USA * Weimin Zheng, Tsinghua University, China Technical Committee * Hasan Abbasi, Oak Ridge National Labs, USA * Tim Armstrong, University of Chicago, USA * Roger Barga, Microsoft, USA * Mihai Budiu, Microsoft Research, USA * Rajkumar Buyya University of Melbourne, Australia * Kyle Chard, University of Chicago, USA * Evangelinos Constantinos, Massachusetts Institute of Technology, USA * Catalin Dumitrescu, Fermi National Labs, USA * Haryadi Gunawi, University of Chicago, USA * Indranil Gupta, University of Illinois at Urbana Champaign, USA * Alexandru Iosup, Delft University of Technology, Netherlands * Florin Isaila, Universidad Carlos III de Madrid, Spain & Argonne National Laboratory, USA * Kamil Iskra, Argonne National Laboratory, USA * Daniel S. Katz, University of Chicago, USA * Jik-Soo Kim, Kristi, Korea * Scott A. Klasky, Oak Ridge National Labs, USA * Mike Lang, Los Alamos National Laboratory, USA * Tonglin Li, Illinois Institute of Technology, USA * Chris Moretti, Princeton University, USA * David O'Hallaron, Carnegie Mellon University, USA * Marlon Pierce, Indiana University, USA * Judy Qiu, Indiana University, USA * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA * Matei Ripeanu, University of British Columbia, Canada * Wei Tang, Argonne National Laboratory, USA * Edward Walker, Whitworth University, USA * Ke Wang, Illinois Institute of Technology, USA * Matthew Woitaszek, Walmart Labs, USA * Rich Wolski, University of California, Santa Barbara, USA * Zhifeng Yun, University of Houston, USA * Zhao Zhang, University of California at Berkeley, USA * Ziming Zheng, University of Chicago, USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= From justinbbt at gmail.com Thu Aug 28 18:49:05 2014 From: justinbbt at gmail.com (Justin bbt) Date: Thu, 28 Aug 2014 19:49:05 -0400 Subject: [Swift-user] running jobs on cluster or cloud Message-ID: Hi all, I could successfully run swift on my local system. Next, I want to use the swift to run some jobs on a cluster. I followed this tutorial. (I am using just a simple cluster- I even could not run the job on one remote node of the cluster) http://swift-lang.org/tutorials/cloud/tutorial.html But, I get this when I run swift p1.swift or other swift Swift 0.94.1 swift-r7114 cog-r3803 RunID: 20140828-1758-ea4phzag Progress: time: Thu, 28 Aug 2014 17:58:15 -0400 Progress: time: Thu, 28 Aug 2014 17:58:24 -0400 Submitted:1 Execution failed: Exception in simulate: Arguments: [] Host: remotehost2 Directory: p1-20140828-1758-ea4phzag/jobs/7/simulate-7k2fxlvl Caused by: Job failed with an exit code of 127 simulation, p1.swift, line 9 --- this is my site.xml file setting 0 10000 /path/to/remote/workdirectory --- if I use this one passive 1 10 10000 .l --- then it loops to my localhost and just repeat submitting the jobs 1. Is this a correct setting? 2. Should I use coaster? I could not understand the description in user guides and documentation about the concepts of coaster and the required setting. Is there any better tutorial which would describe the coaster ? 3. I plan to use the swift later on the cloud (Microsoft Azure). What are the setting required for that? for site.xml and if any other file Thanks in Advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinbbt at gmail.com Thu Aug 28 18:57:40 2014 From: justinbbt at gmail.com (Justin bbt) Date: Thu, 28 Aug 2014 19:57:40 -0400 Subject: [Swift-user] running jobs on cluster or cloud Message-ID: Hi all, > > I could successfully run swift on my local system. > Next, I want to use the swift to run some jobs on a cluster. > > I followed this tutorial. (I am using just a simple cluster- I even could > not run the job on one remote node of the cluster) > http://swift-lang.org/tutorials/cloud/tutorial.html > > But, I get this when I run swift p1.swift or other swift > > Swift 0.94.1 swift-r7114 cog-r3803 > > RunID: 20140828-1758-ea4phzag > Progress: time: Thu, 28 Aug 2014 17:58:15 -0400 > Progress: time: Thu, 28 Aug 2014 17:58:24 -0400 Submitted:1 > Execution failed: > Exception in simulate: > Arguments: [] > Host: remotehost2 > Directory: p1-20140828-1758-ea4phzag/jobs/7/simulate-7k2fxlvl > > Caused by: > Job failed with an exit code of 127 > simulation, p1.swift, line 9 > > > --- this is my site.xml file setting > > > > > 0 > 10000 > /path/to/remote/workdirectory > > > --- if I use this one > > url="myclusterurl" > jobmanager="local:local"/> > passive > 1 > 10 > 10000 > > .l > > --- then it loops to my localhost and just repeat submitting the jobs > > 1. Is this a correct setting? > 2. Should I use coaster? I could not understand the description in user > guides and documentation about the concepts of coaster and the required > setting. Is there any better tutorial which would describe the coaster ? > 3. I plan to use the swift later on the cloud (Microsoft Azure). What are > the setting required for that? for site.xml and if any other file > > > Thanks in Advance. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Fri Aug 29 08:51:55 2014 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Fri, 29 Aug 2014 08:51:55 -0500 Subject: [Swift-user] help with swift scrip In-Reply-To: <1fc0893b3a104865a577fd994dc5cc04@LUCKMAN.anl.gov> References: <1fc0893b3a104865a577fd994dc5cc04@LUCKMAN.anl.gov> Message-ID: Hi Shahram, CC'ing swift-user mailing list, please feel free to subscribe to this mailing list and post for greater visibility and quicker response: http://swift-lang.org/support/index.php Please see my answers inline below: On Thu, Aug 28, 2014 at 5:40 PM, Shahram Mohrehkesh wrote: > Hi Dr. Ketan, > > I just started using Swift for a project to run some image processing > operations on brain images. > > I can run swift successfully on my local system. Next, I followed the > tutorial on the swift website for clusters. However, I cannot run the > swift for running an application with swift on the cluster. I usually get > error "Job failed with an exit code of 127". I guess my main problem is > with my setting in site.xml file. > >From your description, I understand this is a bag of workstations cluster. The exit code 127 seems most likely because Swift is unable to locate the executable. If you are using a tc.data file, make sure that the executable you point to in the third column of this file exists and is executable on the cluster. Having said this, could you indicate the Swift version you are using and the logs. > > 1. Is the coaster the appropriate type for site setting to run on a > cluster? If so, what are the values of parameters? What are the mandatory > parameters? > Yes, coasters are well suited to run on a cluster. The parameters will depend upon the type of cluster you are on: if the nodes are behind a resource manager (eg. PBS, SLURM etc.) or not. Most parameters have default values but I suppose the execution and filesystem providers, handle name, and workdirectory location are mandatory (others on swift-user may correct me). Please see typical sites files attached. Also see this guide to know more about sites file config and examples (probably you already saw it): http://swift-lang.org/guides/trunk/siteguide/siteguide.html > 2. I saw your "Evaluating Cloud Computing for Smart Power Grid Design > Using Parallel Scripting". My cluster is very similar to yours and I am > planning to run my program later on Amazon cloud. So, if you could provide > me your sites.xml file, I would highly appreciate it? > Sites files will differ slightly based on your configuration but find the one we used attached if it helps. Also, we have some tutorials developed for running in Amazon cloud, see this: https://github.com/yadudoc/swift-on-cloud/tree/master/aws > 3. Beside the site.xml, is there any other setting that I should change? > There is some one time configuration that needs to be set. It will depend upon the Swift version you are using and the nature of your environment. > > Regards, > Shahram > -- > > Shahram Mohrehkesh > PhD Candidate > Computer Science > Old Dominion University > Norfolk, VA > http://www.cs.odu.edu/~smohrehk/ > Best Regards, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sites.cloud.xml Type: text/xml Size: 562 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sites.slurm.xml Type: text/xml Size: 1238 bytes Desc: not available URL: From yadudoc1729 at gmail.com Fri Aug 29 10:43:00 2014 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Fri, 29 Aug 2014 10:43:00 -0500 Subject: [Swift-user] running jobs on cluster or cloud In-Reply-To: References: Message-ID: Hi Justin, ??Did you do the following steps: export WORKER_LOCATION="/home/ubuntu" export WORKER_HOSTS=" " export WORKER_USERNAME=ubuntu and then run "source setup.sh" ? When you source the setup.sh scripts you must've gotten a sites.xml and a start-coaster-service.log in your scs folder, could you send us those ? The setup script should start a persistent coaster service and connect to the nodes on amazon, start workers, and generate a sites.xml file that would let your swift scripts run across the amazon nodes. You shouldn't have to make changes to the sites.xml. Alternatively, you could try using the beta release of swift, Swift 0.95 RC6 with the new cloud mechanism: https://github.com/swift-lang/swift-on-cloud/tree/master/aws That will set you up with a headnode on AWS with a few worker nodes that you define, with everything setup to run swift. Thanks, Yadu ? On Thu, Aug 28, 2014 at 6:57 PM, Justin bbt wrote: > > > > Hi all, >> >> I could successfully run swift on my local system. >> Next, I want to use the swift to run some jobs on a cluster. >> >> I followed this tutorial. (I am using just a simple cluster- I even >> could not run the job on one remote node of the cluster) >> http://swift-lang.org/tutorials/cloud/tutorial.html >> >> But, I get this when I run swift p1.swift or other swift >> >> Swift 0.94.1 swift-r7114 cog-r3803 >> >> RunID: 20140828-1758-ea4phzag >> Progress: time: Thu, 28 Aug 2014 17:58:15 -0400 >> Progress: time: Thu, 28 Aug 2014 17:58:24 -0400 Submitted:1 >> Execution failed: >> Exception in simulate: >> Arguments: [] >> Host: remotehost2 >> Directory: p1-20140828-1758-ea4phzag/jobs/7/simulate-7k2fxlvl >> >> Caused by: >> Job failed with an exit code of 127 >> simulation, p1.swift, line 9 >> >> >> --- this is my site.xml file setting >> >> >> > url="myclusteturl"/> >> >> 0 >> 10000 >> /path/to/remote/workdirectory >> >> >> --- if I use this one >> >> > url="myclusterurl" >> jobmanager="local:local"/> >> passive >> 1 >> 10 >> 10000 >> >> .l >> >> --- then it loops to my localhost and just repeat submitting the jobs >> >> 1. Is this a correct setting? >> 2. Should I use coaster? I could not understand the description in user >> guides and documentation about the concepts of coaster and the required >> setting. Is there any better tutorial which would describe the coaster ? >> 3. I plan to use the swift later on the cloud (Microsoft Azure). What are >> the setting required for that? for site.xml and if any other file >> >> >> Thanks in Advance. >> >> > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -- Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: From Matthew.Shaxted at som.com Fri Aug 29 19:48:48 2014 From: Matthew.Shaxted at som.com (Matthew Shaxted) Date: Fri, 29 Aug 2014 20:48:48 -0400 Subject: [Swift-user] Swift Coaster Service & Docker Containers Message-ID: Hi All, I'm trying to find a way to run Swift workflows on multi-host docker containers, and wondering if anybody has had success with this. When I start a docker container and define specific ports to open on the container, they mapped to random ports on the host machine. So for example, I can start a container with an ssh port open from a host "10.1.1.1", and can then access this container across hosts with "ssh compute at 10.1.1.1 -p 49160". Now I'm hoping to link these docker containers to Swift's start-coaster-service. I think it would be possible and relatively easy if I can say, start the coasters on a series of IP addresses AND ports. So a host's file perhaps would look something like below, and coasters would be started on the correct docker container: 10.1.1.1: 49160 10.1.1.2: 34155 10.1.1.2: 34156 ... Does this make sense? Is it possible to start coasters by specifying an IP address and port number? Any thoughts are greatly appreciated. Thanks, Matthew MATTHEW SHAXTED SKIDMORE, OWINGS & MERRILL LLP 224 SOUTH MICHIGAN AVENUE CHICAGO, IL 60604 T (312) 360-4368 MATTHEW.SHAXTED at SOM.COM [cid:image004.png at 01CFC3C2.402B1230] The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. [cid:image003.gif at 01CFC3AA.BD34BB40] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.gif Type: image/gif Size: 566 bytes Desc: image003.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.png Type: image/png Size: 5311 bytes Desc: image004.png URL: From wilde at anl.gov Fri Aug 29 20:34:04 2014 From: wilde at anl.gov (Michael Wilde) Date: Fri, 29 Aug 2014 20:34:04 -0500 Subject: [Swift-user] Swift Coaster Service & Docker Containers In-Reply-To: References: Message-ID: <54012A0C.4070703@anl.gov> Matthew, I would treat this case similar to the configuration you'd use for a set of virtual machines. Start one coaster service for each pool of identical docker containers that you want to run. For each pool of containers, run a Swift worker (worker.pl) in the container and have it connect back to the coaster service you designate to manage that pool. I'm assuming that from a docker contained you can connect out to any reachable IP address. Another approach is to treat the containers like a set of ad-hoc compute nodes, and ssh into them with automatic coasters using the ssh:local jobmanager setting. We'll try to test and publish a config for such cases. We'd welcome your help with that. - Mike On 8/29/14, 7:48 PM, Matthew Shaxted wrote: > > Hi All, > > I'm trying to find a way to run Swift workflows on multi-host docker > containers, and wondering if anybody has had success with this. > > When I start a docker container and define specific ports to open on > the container, they mapped to random ports on the host machine. So for > example, I can start a container with an ssh port open from a host > "10.1.1.1", and can then access this container across hosts with "ssh > compute at 10.1.1.1 --p 49160". > > Now I'm hoping to link these docker containers to Swift's > start-coaster-service. I think it would be possible and relatively > easy if I can say, start the coasters on a series of IP addresses AND > ports. So a host's file perhaps would look something like below, and > coasters would be started on the correct docker container: > > 10.1.1.1: 49160 > > 10.1.1.2: 34155 > > 10.1.1.2: 34156 ... > > Does this make sense? Is it possible to start coasters by specifying > an IP address and port number? > > Any thoughts are greatly appreciated. > > Thanks, > > Matthew > > MATTHEW SHAXTED > > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > cid:image001.png at 01CF9071.6FB46030 > > The information contained in this communication may be confidential, > is intended only for the use of the recipient(s) named above, and may > be legally privileged. If the reader of this message is not the > intended recipient, you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of its > contents, is strictly prohibited and may be unlawful. If you have > received this communication in error, please return it to the > sen???der immediately and delete the original message and any copy of > it from your computer system. If you have any questions concerning > this message, please contact the sender. > > http://intranet.som.com/common/admin/file.cfm?f=%2Fresources%2Fcontent%2F5%2F0%2F4%2F4%2F6%2F4%2F0%2F3%2Fdocuments%2Fimagea560bf%2Egif%406e10073b%2E30854c37 > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 5311 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 566 bytes Desc: not available URL: From yadudoc1729 at gmail.com Fri Aug 29 21:33:21 2014 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Fri, 29 Aug 2014 21:33:21 -0500 Subject: [Swift-user] Swift Coaster Service & Docker Containers In-Reply-To: <54012A0C.4070703@anl.gov> References: <54012A0C.4070703@anl.gov> Message-ID: Hi Matthew, You can use the coaster-service directly with the following arguments : coaster-service -p $SERVICEPORT -localport $WORKERPORT -nosec -passive &> /var/log/coaster-service.logs The workers running under docker should be connecting to the WORKERPORT and the swift client would use the SERVICEPORT. Another thing to remember is to make sure that you use the the coaster-service and worker.pl from the same swift release. The start-coaster-service command is a wrapper over coaster-service which in addition to starting the coaster service, also tries to ssh the worker.pl to the WORKER_HOSTS. In the situation that you describe, I think it makes sense to have the workers just connect back to the coaster-service which is listening on a known IP / port. Thanks, Yadu On Fri, Aug 29, 2014 at 8:34 PM, Michael Wilde wrote: > Matthew, > > I would treat this case similar to the configuration you'd use for a set > of virtual machines. > > Start one coaster service for each pool of identical docker containers > that you want to run. > > For each pool of containers, run a Swift worker (worker.pl) in the > container and have it connect back to the coaster service you designate to > manage that pool. > > I'm assuming that from a docker contained you can connect out to any > reachable IP address. > > Another approach is to treat the containers like a set of ad-hoc compute > nodes, and ssh into them with automatic coasters using the ssh:local > jobmanager setting. > > We'll try to test and publish a config for such cases. We'd welcome your > help with that. > > - Mike > > > > On 8/29/14, 7:48 PM, Matthew Shaxted wrote: > > Hi All, > > > > I?m trying to find a way to run Swift workflows on multi-host docker > containers, and wondering if anybody has had success with this. > > > > When I start a docker container and define specific ports to open on the > container, they mapped to random ports on the host machine. So for example, > I can start a container with an ssh port open from a host ?10.1.1.1?, and > can then access this container across hosts with ?ssh compute at 10.1.1.1 ?p > 49160?. > > > > Now I?m hoping to link these docker containers to Swift?s > start-coaster-service. I think it would be possible and relatively easy if > I can say, start the coasters on a series of IP addresses AND ports. So a > host?s file perhaps would look something like below, and coasters would be > started on the correct docker container: > > > > 10.1.1.1: 49160 > > 10.1.1.2: 34155 > > 10.1.1.2: 34156 ? > > > > Does this make sense? Is it possible to start coasters by specifying an IP > address and port number? > > > > Any thoughts are greatly appreciated. > > > > Thanks, > > Matthew > > > > > > MATTHEW SHAXTED > > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > > > [image: cid:image001.png at 01CF9071.6FB46030] > > The information contained in this communication may be confidential, is > intended only for the use of the recipient(s) named above, and may be > legally privileged. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, distribution, or > copying of this communication, or any of its contents, is strictly > prohibited and may be unlawful. If you have received this communication in > error, please return it to the sen???der immediately and delete the > original message and any copy of it from your computer system. If you have > any questions concerning this message, please contact the sender. > > > > [image: > http://intranet.som.com/common/admin/file.cfm?f=%2Fresources%2Fcontent%2F5%2F0%2F4%2F4%2F6%2F4%2F0%2F3%2Fdocuments%2Fimagea560bf%2Egif%406e10073b%2E30854c37] > > > > > _______________________________________________ > Swift-user mailing listSwift-user at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -- Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 566 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 5311 bytes Desc: not available URL: From Matthew.Shaxted at som.com Sat Aug 30 08:39:38 2014 From: Matthew.Shaxted at som.com (Matthew Shaxted) Date: Sat, 30 Aug 2014 09:39:38 -0400 Subject: [Swift-user] Swift Coaster Service & Docker Containers In-Reply-To: References: <54012A0C.4070703@anl.gov> Message-ID: <41C141B3-570C-43A7-A30C-D6740B38E829@som.com> Thanks Mike and Yadu, Yes it would be great if there is a published config for this use case, and I am willing to help in whatever capacity I can. The docker containers can reach out to external IP addresses, so as you both mention the best way may be to run the worker.pl in each container and have it connect back to the coaster service. I start the coaster service on a known ip/port host using the command you specify. I think I conceptually understand how this could work and will try to test something. I have an ubuntu docker container configured to start ssh services at the command below. It sounds like, however, if I can run the worker.pl on each container and connect to the coaster service, ssh is not even required. docker run -i -p 22 -t mattshax/precise_ep_node sh -c '/usr/sbin/sshd -D' On Aug 29, 2014, at 9:33 PM, "Yadu Nand" > wrote: Hi Matthew, You can use the coaster-service directly with the following arguments : coaster-service -p $SERVICEPORT -localport $WORKERPORT -nosec -passive &> /var/log/coaster-service.logs The workers running under docker should be connecting to the WORKERPORT and the swift client would use the SERVICEPORT. Another thing to remember is to make sure that you use the the coaster-service and worker.pl from the same swift release. The start-coaster-service command is a wrapper over coaster-service which in addition to starting the coaster service, also tries to ssh the worker.pl to the WORKER_HOSTS. In the situation that you describe, I think it makes sense to have the workers just connect back to the coaster-service which is listening on a known IP / port. Thanks, Yadu On Fri, Aug 29, 2014 at 8:34 PM, Michael Wilde > wrote: Matthew, I would treat this case similar to the configuration you'd use for a set of virtual machines. Start one coaster service for each pool of identical docker containers that you want to run. For each pool of containers, run a Swift worker (worker.pl) in the container and have it connect back to the coaster service you designate to manage that pool. I'm assuming that from a docker contained you can connect out to any reachable IP address. Another approach is to treat the containers like a set of ad-hoc compute nodes, and ssh into them with automatic coasters using the ssh:local jobmanager setting. We'll try to test and publish a config for such cases. We'd welcome your help with that. - Mike On 8/29/14, 7:48 PM, Matthew Shaxted wrote: Hi All, I?m trying to find a way to run Swift workflows on multi-host docker containers, and wondering if anybody has had success with this. When I start a docker container and define specific ports to open on the container, they mapped to random ports on the host machine. So for example, I can start a container with an ssh port open from a host ?10.1.1.1?, and can then access this container across hosts with ?ssh compute at 10.1.1.1 ?p 49160?. Now I?m hoping to link these docker containers to Swift?s start-coaster-service. I think it would be possible and relatively easy if I can say, start the coasters on a series of IP addresses AND ports. So a host?s file perhaps would look something like below, and coasters would be started on the correct docker container: 10.1.1.1: 49160 10.1.1.2: 34155 10.1.1.2: 34156 ? Does this make sense? Is it possible to start coasters by specifying an IP address and port number? Any thoughts are greatly appreciated. Thanks, Matthew MATTHEW SHAXTED SKIDMORE, OWINGS & MERRILL LLP 224 SOUTH MICHIGAN AVENUE CHICAGO, IL 60604 T (312) 360-4368 MATTHEW.SHAXTED at SOM.COM The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sen???der immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Yadu Nand B _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinbbt at gmail.com Sat Aug 30 00:28:30 2014 From: justinbbt at gmail.com (Justin bbt) Date: Sat, 30 Aug 2014 01:28:30 -0400 Subject: [Swift-user] running jobs on cluster or cloud In-Reply-To: References: Message-ID: For cluster: When I run the start-caoster-service, I receive the following, in which it asks for password and then says Permission is denied Start-coaster-service... Configuration: /home/lenovo/swift-cloud-tutorial/scs/coaster-service.conf Service address: localhost Starting coaster-service Service port: 52809 Local port: 58460 Generating sites.xml username at ipadress's password: username at ipadress's password: Starting worker on username@ lenovo at lenovo-laptop:~/swift-cloud-tutorial/scs$username at ipadress's password: Permission denied, please try again. username at ipadress's password: Permission denied, please try again. username at ipadress's password: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). This happens though I have created my keys with ssh-keygen. (only changed that I made was to create rsa keys rather than dsa keys - my cluster did not accept dsa). I can connect with rsa keygen and my passphrase for regular ssh The output of my sites.xml from this partial running of start-coaster-service is passive 1 10 10000 . Using this XML , I just get a sequence of job submission every 30 seconds, no finished jobs. BTW, I have a public ip for my cluster and then each compute node has a local/private ip. In export WORKER_HOSTS=" " currently I just set the public IP address which still I am not successful with this one node even. I was wondering how should I set the other IPs? Does it mean that I have to install swift on the cluster? I will look at the new release of swift for AWS. Thanks, J. On Fri, Aug 29, 2014 at 11:43 AM, Yadu Nand wrote: > Hi Justin, > > ??Did you do the following steps: > export WORKER_LOCATION="/home/ubuntu" > export WORKER_HOSTS=" " > export WORKER_USERNAME=ubuntu > > and then run "source setup.sh" ? > When you source the setup.sh scripts you must've gotten a sites.xml and a > start-coaster-service.log in your scs folder, could you send us those ? > The setup script should start a persistent coaster service and connect to > the nodes on amazon, start workers, and generate a sites.xml file > that would let your swift scripts run across the amazon nodes. You > shouldn't have to make changes to the sites.xml. > > Alternatively, you could try using the beta release of swift, Swift 0.95 > RC6 with the new cloud mechanism: > https://github.com/swift-lang/swift-on-cloud/tree/master/aws > > That will set you up with a headnode on AWS with a few worker nodes that > you define, with everything setup to run swift. > > > Thanks, > Yadu > ? > > > On Thu, Aug 28, 2014 at 6:57 PM, Justin bbt wrote: > >> >> >> >> Hi all, >>> >>> I could successfully run swift on my local system. >>> Next, I want to use the swift to run some jobs on a cluster. >>> >>> I followed this tutorial. (I am using just a simple cluster- I even >>> could not run the job on one remote node of the cluster) >>> http://swift-lang.org/tutorials/cloud/tutorial.html >>> >>> But, I get this when I run swift p1.swift or other swift >>> >>> Swift 0.94.1 swift-r7114 cog-r3803 >>> >>> RunID: 20140828-1758-ea4phzag >>> Progress: time: Thu, 28 Aug 2014 17:58:15 -0400 >>> Progress: time: Thu, 28 Aug 2014 17:58:24 -0400 Submitted:1 >>> Execution failed: >>> Exception in simulate: >>> Arguments: [] >>> Host: remotehost2 >>> Directory: p1-20140828-1758-ea4phzag/jobs/7/simulate-7k2fxlvl >>> >>> Caused by: >>> Job failed with an exit code of 127 >>> simulation, p1.swift, line 9 >>> >>> >>> --- this is my site.xml file setting >>> >>> >>> >> url="myclusteturl"/> >>> >>> 0 >>> 10000 >>> /path/to/remote/workdirectory >>> >>> >>> --- if I use this one >>> >>> >> url="myclusterurl" >>> jobmanager="local:local"/> >>> passive >>> 1 >>> 10 >>> 10000 >>> >>> .l >>> >>> --- then it loops to my localhost and just repeat submitting the jobs >>> >>> 1. Is this a correct setting? >>> 2. Should I use coaster? I could not understand the description in user >>> guides and documentation about the concepts of coaster and the required >>> setting. Is there any better tutorial which would describe the coaster ? >>> 3. I plan to use the swift later on the cloud (Microsoft Azure). What >>> are the setting required for that? for site.xml and if any other file >>> >>> >>> Thanks in Advance. >>> >>> >> >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >> > > > > -- > Yadu Nand B > > -------------- next part -------------- An HTML attachment was scrubbed... URL: