From yadunand at uchicago.edu Mon Nov 3 10:14:28 2014 From: yadunand at uchicago.edu (Yadu Nand Babuji) Date: Mon, 03 Nov 2014 10:14:28 -0600 Subject: [Swift-devel] Tryswift is down Message-ID: <5457A9E4.7080502@uchicago.edu> Hi, Sophia reported yesterday that the tryswift webpages are not working properly. I cleaned up old files, and have been able to run on swiftvm2 and vm3. vm4 is not responding at all. I can set up a few checks to ensure that the webservers on all vms are reachable. This does not address the situation where the webpages load but the buttons do not respond, or the swift runs do not complete in a reasonable amount of time, both of which are harder to test. Thanks, Yadu From wilde at anl.gov Sat Nov 8 12:17:01 2014 From: wilde at anl.gov (Michael Wilde) Date: Sat, 8 Nov 2014 12:17:01 -0600 Subject: [Swift-devel] Example of wrapper staging? Message-ID: <545E5E1D.8000702@anl.gov> Does anyone have a working example of wrapper staging? I'm hoping to run this on trunk but can call back to older releases if needed. Thanks, - Mike From hategan at mcs.anl.gov Sat Nov 8 16:13:27 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 8 Nov 2014 14:13:27 -0800 Subject: [Swift-devel] Example of wrapper staging? In-Reply-To: <545E5E1D.8000702@anl.gov> References: <545E5E1D.8000702@anl.gov> Message-ID: <1415484807.975.10.camel@echo> Hi, Some changes were needed since the wrapper staging code wasn't updated in a while. It should work now (rev. 6172). Here's a simple example: -------------------------------------------------------- type file; app (file c) cat(file a, file b) { cat filename(a) filename(b) stdout=filename(c); } file c <"c.txt">; file a <"a.txt">; file b <"http://mirror.internode.on.net/pub/test/1meg.test">; c = cat(a, b); -------------------------------------------------------- You would need a proper swift.conf. Here's an example of that: -------------------------------------------------------- site.local { staging: "wrapper" scratch: "/some/scratch/dir" # *see email comments for details on the following option # wrapperStagingLocalServer: "file://" } -------------------------------------------------------- With a proper scratch dir there, this *should* work for you without any additional tweaking. Check _swiftwrap.wrapperstaging's stagein() and stageout() functions. That's where you would add additional protocols. The wrapperStagingLocalServer sets the default staging location for things that do not have an explicit scheme. In the case above, "c.txt" and "a.txt" are staged in and out using "file://". If you wanted the default to be some gridftp server, you would say something like 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course, _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs, which it doesn't at the moment. Mihael On Sat, 2014-11-08 at 12:17 -0600, Michael Wilde wrote: > Does anyone have a working example of wrapper staging? > > I'm hoping to run this on trunk but can call back to older releases if > needed. > > Thanks, > > - Mike > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From wilde at anl.gov Sat Nov 8 16:37:30 2014 From: wilde at anl.gov (Michael Wilde) Date: Sat, 8 Nov 2014 16:37:30 -0600 Subject: [Swift-devel] Example of wrapper staging? In-Reply-To: <1415484807.975.10.camel@echo> References: <545E5E1D.8000702@anl.gov> <1415484807.975.10.camel@echo> Message-ID: <545E9B2A.1020309@anl.gov> I get this error: org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected property 'site.local.wrapperStagingLocalServer' at org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214) Ive tried spelling this property both as in your sample .conf and as in your text: wrapperStagingLocalServer and wrapperStageingLocalServer Neither is accepted. Did a schema change not get committed? For general info: where is the json schema file that a user can check to find all valid conf parameters? Thanks, - Mike On 11/8/14 4:13 PM, Mihael Hategan wrote: > type file; > > app (file c) cat(file a, file b) { > cat filename(a) filename(b) stdout=filename(c); > } > > file c <"c.txt">; > file a <"a.txt">; > file b <"http://mirror.internode.on.net/pub/test/1meg.test">; > > c = cat(a, b); > -------------------------------------------------------- > > You would need a proper swift.conf. Here's an example of that: > -------------------------------------------------------- > site.local { > staging: "wrapper" > scratch: "/some/scratch/dir" > # *see email comments for details on the following option > # wrapperStagingLocalServer:"file://" > } > -------------------------------------------------------- > > > With a proper scratch dir there, this*should* work for you without any > additional tweaking. > > Check _swiftwrap.wrapperstaging's stagein() and stageout() functions. > That's where you would add additional protocols. > > The wrapperStagingLocalServer sets the default staging location for > things that do not have an explicit scheme. In the case above, "c.txt" > and "a.txt" are staged in and out using"file://". If you wanted the > default to be some gridftp server, you would say something like > 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course, > _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs, > which it doesn't at the moment. > > Mihael -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sat Nov 8 16:41:49 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 8 Nov 2014 14:41:49 -0800 Subject: [Swift-devel] Example of wrapper staging? In-Reply-To: <545E9B2A.1020309@anl.gov> References: <545E5E1D.8000702@anl.gov> <1415484807.975.10.camel@echo> <545E9B2A.1020309@anl.gov> Message-ID: <1415486509.1864.2.camel@echo> (solved on IM but replying here for reference). I made a mistake in the swift.conf I sent. wrapperStagingLocalServer is a global option not a site option. However, it should work with the vanilla swift.conf posted, which had the wrong option commented out. Mihael On Sat, 2014-11-08 at 16:37 -0600, Michael Wilde wrote: > I get this error: > > org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected > property 'site.local.wrapperStagingLocalServer' > at > org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214) > > Ive tried spelling this property both as in your sample .conf and as in > your text: > > wrapperStagingLocalServer and wrapperStageingLocalServer > > Neither is accepted. Did a schema change not get committed? > > For general info: where is the json schema file that a user can check to > find all valid conf parameters? > > Thanks, > > - Mike > > > > On 11/8/14 4:13 PM, Mihael Hategan wrote: > > type file; > > > > app (file c) cat(file a, file b) { > > cat filename(a) filename(b) stdout=filename(c); > > } > > > > file c <"c.txt">; > > file a <"a.txt">; > > file b <"http://mirror.internode.on.net/pub/test/1meg.test">; > > > > c = cat(a, b); > > -------------------------------------------------------- > > > > You would need a proper swift.conf. Here's an example of that: > > -------------------------------------------------------- > > site.local { > > staging: "wrapper" > > scratch: "/some/scratch/dir" > > # *see email comments for details on the following option > > # wrapperStagingLocalServer:"file://" > > } > > -------------------------------------------------------- > > > > > > With a proper scratch dir there, this*should* work for you without any > > additional tweaking. > > > > Check _swiftwrap.wrapperstaging's stagein() and stageout() functions. > > That's where you would add additional protocols. > > > > The wrapperStagingLocalServer sets the default staging location for > > things that do not have an explicit scheme. In the case above, "c.txt" > > and "a.txt" are staged in and out using"file://". If you wanted the > > default to be some gridftp server, you would say something like > > 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course, > > _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs, > > which it doesn't at the moment. > > > > Mihael > From yadunand at uchicago.edu Sat Nov 8 16:45:45 2014 From: yadunand at uchicago.edu (Yadu Nand Babuji) Date: Sat, 08 Nov 2014 16:45:45 -0600 Subject: [Swift-devel] Example of wrapper staging? In-Reply-To: <545E9B2A.1020309@anl.gov> References: <545E5E1D.8000702@anl.gov> <1415484807.975.10.camel@echo> <545E9B2A.1020309@anl.gov> Message-ID: <545E9D19.6000205@uchicago.edu> Hi Mike, The schema is here swift-k/resources/swift.conf.schema, and I do see this wrapperStagingLocalServer defined in the schema. Here's a github link to the schema : https://github.com/swift-lang/swift-k/blob/master/resources/swift.conf.schema -Yadu On 11/08/2014 04:37 PM, Michael Wilde wrote: > I get this error: > > org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected > property 'site.local.wrapperStagingLocalServer' > at > org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214) > > Ive tried spelling this property both as in your sample .conf and as > in your text: > > wrapperStagingLocalServer and wrapperStageingLocalServer > > Neither is accepted. Did a schema change not get committed? > > For general info: where is the json schema file that a user can check > to find all valid conf parameters? > > Thanks, > > - Mike > > > > On 11/8/14 4:13 PM, Mihael Hategan wrote: >> type file; >> >> app (file c) cat(file a, file b) { >> cat filename(a) filename(b) stdout=filename(c); >> } >> >> file c <"c.txt">; >> file a <"a.txt">; >> file b <"http://mirror.internode.on.net/pub/test/1meg.test">; >> >> c = cat(a, b); >> -------------------------------------------------------- >> >> You would need a proper swift.conf. Here's an example of that: >> -------------------------------------------------------- >> site.local { >> staging: "wrapper" >> scratch: "/some/scratch/dir" >> # *see email comments for details on the following option >> # wrapperStagingLocalServer:"file://" >> } >> -------------------------------------------------------- >> >> >> With a proper scratch dir there, this*should* work for you without any >> additional tweaking. >> >> Check _swiftwrap.wrapperstaging's stagein() and stageout() functions. >> That's where you would add additional protocols. >> >> The wrapperStagingLocalServer sets the default staging location for >> things that do not have an explicit scheme. In the case above, "c.txt" >> and "a.txt" are staged in and out using"file://". If you wanted the >> default to be some gridftp server, you would say something like >> 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course, >> _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs, >> which it doesn't at the moment. >> >> Mihael > > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Sat Nov 8 17:32:37 2014 From: wilde at anl.gov (Michael Wilde) Date: Sat, 8 Nov 2014 17:32:37 -0600 Subject: [Swift-devel] Example of wrapper staging? In-Reply-To: <1415486509.1864.2.camel@echo> References: <545E5E1D.8000702@anl.gov> <1415484807.975.10.camel@echo> <545E9B2A.1020309@anl.gov> <1415486509.1864.2.camel@echo> Message-ID: <545EA815.7020602@anl.gov> Ok, that works great now, both with the conf you posted and with this one: wrapperStagingLocalServer: "file://" site.local { staging: "wrapper" scratch: "/tmp/wilde/swiftest" } Thanks, - Mike On 11/8/14 4:41 PM, Mihael Hategan wrote: > (solved on IM but replying here for reference). > > I made a mistake in the swift.conf I sent. wrapperStagingLocalServer is > a global option not a site option. However, it should work with the > vanilla swift.conf posted, which had the wrong option commented out. > > Mihael > > > On Sat, 2014-11-08 at 16:37 -0600, Michael Wilde wrote: >> I get this error: >> >> org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected >> property 'site.local.wrapperStagingLocalServer' >> at >> org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214) >> >> Ive tried spelling this property both as in your sample .conf and as in >> your text: >> >> wrapperStagingLocalServer and wrapperStageingLocalServer >> >> Neither is accepted. Did a schema change not get committed? >> >> For general info: where is the json schema file that a user can check to >> find all valid conf parameters? >> >> Thanks, >> >> - Mike >> >> >> >> On 11/8/14 4:13 PM, Mihael Hategan wrote: >>> type file; >>> >>> app (file c) cat(file a, file b) { >>> cat filename(a) filename(b) stdout=filename(c); >>> } >>> >>> file c <"c.txt">; >>> file a <"a.txt">; >>> file b <"http://mirror.internode.on.net/pub/test/1meg.test">; >>> >>> c = cat(a, b); >>> -------------------------------------------------------- >>> >>> You would need a proper swift.conf. Here's an example of that: >>> -------------------------------------------------------- >>> site.local { >>> staging: "wrapper" >>> scratch: "/some/scratch/dir" >>> # *see email comments for details on the following option >>> # wrapperStagingLocalServer:"file://" >>> } >>> -------------------------------------------------------- >>> >>> >>> With a proper scratch dir there, this*should* work for you without any >>> additional tweaking. >>> >>> Check _swiftwrap.wrapperstaging's stagein() and stageout() functions. >>> That's where you would add additional protocols. >>> >>> The wrapperStagingLocalServer sets the default staging location for >>> things that do not have an explicit scheme. In the case above, "c.txt" >>> and "a.txt" are staged in and out using"file://". If you wanted the >>> default to be some gridftp server, you would say something like >>> 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course, >>> _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs, >>> which it doesn't at the moment. >>> >>> Mihael > -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wilde at anl.gov Sun Nov 9 14:05:09 2014 From: wilde at anl.gov (Michael Wilde) Date: Sun, 9 Nov 2014 14:05:09 -0600 Subject: [Swift-devel] Problems Re: Example of wrapper staging? In-Reply-To: <545EA815.7020602@anl.gov> References: <545E5E1D.8000702@anl.gov> <1415484807.975.10.camel@echo> <545E9B2A.1020309@anl.gov> <1415486509.1864.2.camel@echo> <545EA815.7020602@anl.gov> Message-ID: <545FC8F5.5000007@anl.gov> Oddly, when I use "sh" -> /bin/sh for the app, this example fails. The swift run succeeds, but the returned file is zero-length (for the example below) or contains a single newline (when I test a sh -c of a simple echo). It *seems* that using cat as the app works while /bin/sh fails to return output correctly. I'll need to debug further, but below are the working and then failing swift scripts: # This works, c.txt comes back as 1M+12 bytes: $ cat wrapstage.swift type file; app (file c) cat(file a, file b) { cat filename(a) filename(b) stdout=filename(c); } file c <"c.txt">; file a <"a.txt">; file b <"http://mirror.internode.on.net/pub/test/1meg.test">; c = cat(a, b); # this fails: c.txt comes back as zero length: $ cat wrapstage2.swift type file; app (file c) cat(file a, file b) { sh "-c" strcat("/bin/cat ", filename(a), " ", filename(b)) stdout=filename(c); } file c <"c.txt">; file a <"a.txt">; file b <"http://mirror.internode.on.net/pub/test/1meg.test">; c = cat(a, b); swift$ A bug, or user error? - Mike On 11/8/14 5:32 PM, Michael Wilde wrote: > Ok, that works great now, both with the conf you posted and with this one: > > wrapperStagingLocalServer: "file://" > > site.local { > staging: "wrapper" > scratch: "/tmp/wilde/swiftest" > } > > Thanks, > > - Mike > > On 11/8/14 4:41 PM, Mihael Hategan wrote: >> (solved on IM but replying here for reference). >> >> I made a mistake in the swift.conf I sent. wrapperStagingLocalServer is >> a global option not a site option. However, it should work with the >> vanilla swift.conf posted, which had the wrong option commented out. >> >> Mihael >> >> >> On Sat, 2014-11-08 at 16:37 -0600, Michael Wilde wrote: >>> I get this error: >>> >>> org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected >>> property 'site.local.wrapperStagingLocalServer' >>> at >>> org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214) >>> >>> Ive tried spelling this property both as in your sample .conf and as in >>> your text: >>> >>> wrapperStagingLocalServer and wrapperStageingLocalServer >>> >>> Neither is accepted. Did a schema change not get committed? >>> >>> For general info: where is the json schema file that a user can check to >>> find all valid conf parameters? >>> >>> Thanks, >>> >>> - Mike >>> >>> >>> >>> On 11/8/14 4:13 PM, Mihael Hategan wrote: >>>> type file; >>>> >>>> app (file c) cat(file a, file b) { >>>> cat filename(a) filename(b) stdout=filename(c); >>>> } >>>> >>>> file c <"c.txt">; >>>> file a <"a.txt">; >>>> file b <"http://mirror.internode.on.net/pub/test/1meg.test">; >>>> >>>> c = cat(a, b); >>>> -------------------------------------------------------- >>>> >>>> You would need a proper swift.conf. Here's an example of that: >>>> -------------------------------------------------------- >>>> site.local { >>>> staging: "wrapper" >>>> scratch: "/some/scratch/dir" >>>> # *see email comments for details on the following option >>>> # wrapperStagingLocalServer:"file://" >>>> } >>>> -------------------------------------------------------- >>>> >>>> >>>> With a proper scratch dir there, this*should* work for you without any >>>> additional tweaking. >>>> >>>> Check _swiftwrap.wrapperstaging's stagein() and stageout() functions. >>>> That's where you would add additional protocols. >>>> >>>> The wrapperStagingLocalServer sets the default staging location for >>>> things that do not have an explicit scheme. In the case above, "c.txt" >>>> and "a.txt" are staged in and out using"file://". If you wanted the >>>> default to be some gridftp server, you would say something like >>>> 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course, >>>> _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs, >>>> which it doesn't at the moment. >>>> >>>> Mihael -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wilde at anl.gov Sun Nov 9 14:25:13 2014 From: wilde at anl.gov (Michael Wilde) Date: Sun, 9 Nov 2014 14:25:13 -0600 Subject: [Swift-devel] Problems Re: Example of wrapper staging? In-Reply-To: <545FC8F5.5000007@anl.gov> References: <545E5E1D.8000702@anl.gov> <1415484807.975.10.camel@echo> <545E9B2A.1020309@anl.gov> <1415486509.1864.2.camel@echo> <545EA815.7020602@anl.gov> <545FC8F5.5000007@anl.gov> Message-ID: <545FCDA9.9030007@anl.gov> Below is a simpler example of the failing case: $ hostname midway001 # Works with echo as app and wrapperStaging: $ cat t03.swift type file; app (file o) echo () { echo "hello" stdout=filename(o); } file o <"out.txt">; o = echo(); $ cat swift.conf wrapperStagingLocalServer: "file://" app.sh { executable: /bin/sh, maxWallTime: "00:01:00" } site.local { staging: "wrapper" scratch: "/tmp/wilde/swiftest" workDirectory: "/tmp/wilde/swiftwork" } lazyErrors: false executionRetries: 0 keepSiteDir: true alwaysTransferWrapperLog: true $ rm out.txt # Fails with sh as app and wrapper staging: $ swift -config swift.conf t03.swift Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 heads/master 6172 RunID: run011 Progress: Sun, 09 Nov 2014 20:20:39+0000 Final status:Sun, 09 Nov 2014 20:20:40+0000 Finished successfully:1 $ cat out.txt hello $ cat t04.swift type file; app (file o) echo () { sh "-c" "/bin/echo hello" stdout=filename(o); } file o <"out.txt">; o = echo(); $ swift -config swift.conf t04.swift Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 heads/master 6172 RunID: run012 Progress: Sun, 09 Nov 2014 20:21:09+0000 Final status:Sun, 09 Nov 2014 20:21:09+0000 Finished successfully:1 $ cat out.txt $ ls -l out.txt -rw-rw-r-- 1 wilde wilde 1 Nov 9 14:21 out.txt $ od -x out.txt 0000000 000a 0000001 # Works with sh as app and default staging: $ swift -config swift.conf.nows t04.swift Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 heads/master 6172 RunID: run013 Progress: Sun, 09 Nov 2014 20:21:59+0000 Final status:Sun, 09 Nov 2014 20:21:59+0000 Finished successfully:1 $ cat swift.conf.nows wrapperStagingLocalServer: "file://" app.sh { executable: /bin/sh, maxWallTime: "00:01:00" } lazyErrors: false executionRetries: 0 $ cat out.txt hello $ On 11/9/14 2:05 PM, Michael Wilde wrote: > Oddly, when I use "sh" -> /bin/sh for the app, this example fails. The > swift run succeeds, but the returned file is zero-length (for the > example below) or contains a single newline (when I test a sh -c of a > simple echo). > > It *seems* that using cat as the app works while /bin/sh fails to return > output correctly. > > I'll need to debug further, but below are the working and then failing > swift scripts: > > # This works, c.txt comes back as 1M+12 bytes: > > $ cat wrapstage.swift > type file; > > app (file c) cat(file a, file b) { > cat filename(a) filename(b) stdout=filename(c); > } > > file c <"c.txt">; > file a <"a.txt">; > file b <"http://mirror.internode.on.net/pub/test/1meg.test">; > > c = cat(a, b); > > # this fails: c.txt comes back as zero length: > > $ cat wrapstage2.swift > type file; > > app (file c) cat(file a, file b) { > sh "-c" strcat("/bin/cat ", filename(a), " ", filename(b)) > stdout=filename(c); > } > > file c <"c.txt">; > file a <"a.txt">; > file b <"http://mirror.internode.on.net/pub/test/1meg.test">; > > c = cat(a, b); > swift$ > > A bug, or user error? > > - Mike > > On 11/8/14 5:32 PM, Michael Wilde wrote: >> Ok, that works great now, both with the conf you posted and with this one: >> >> wrapperStagingLocalServer: "file://" >> >> site.local { >> staging: "wrapper" >> scratch: "/tmp/wilde/swiftest" >> } >> >> Thanks, >> >> - Mike >> >> On 11/8/14 4:41 PM, Mihael Hategan wrote: >>> (solved on IM but replying here for reference). >>> >>> I made a mistake in the swift.conf I sent. wrapperStagingLocalServer is >>> a global option not a site option. However, it should work with the >>> vanilla swift.conf posted, which had the wrong option commented out. >>> >>> Mihael >>> >>> >>> On Sat, 2014-11-08 at 16:37 -0600, Michael Wilde wrote: >>>> I get this error: >>>> >>>> org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected >>>> property 'site.local.wrapperStagingLocalServer' >>>> at >>>> org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214) >>>> >>>> Ive tried spelling this property both as in your sample .conf and as in >>>> your text: >>>> >>>> wrapperStagingLocalServer and wrapperStageingLocalServer >>>> >>>> Neither is accepted. Did a schema change not get committed? >>>> >>>> For general info: where is the json schema file that a user can check to >>>> find all valid conf parameters? >>>> >>>> Thanks, >>>> >>>> - Mike >>>> >>>> >>>> >>>> On 11/8/14 4:13 PM, Mihael Hategan wrote: >>>>> type file; >>>>> >>>>> app (file c) cat(file a, file b) { >>>>> cat filename(a) filename(b) stdout=filename(c); >>>>> } >>>>> >>>>> file c <"c.txt">; >>>>> file a <"a.txt">; >>>>> file b <"http://mirror.internode.on.net/pub/test/1meg.test">; >>>>> >>>>> c = cat(a, b); >>>>> -------------------------------------------------------- >>>>> >>>>> You would need a proper swift.conf. Here's an example of that: >>>>> -------------------------------------------------------- >>>>> site.local { >>>>> staging: "wrapper" >>>>> scratch: "/some/scratch/dir" >>>>> # *see email comments for details on the following option >>>>> # wrapperStagingLocalServer:"file://" >>>>> } >>>>> -------------------------------------------------------- >>>>> >>>>> >>>>> With a proper scratch dir there, this*should* work for you without any >>>>> additional tweaking. >>>>> >>>>> Check _swiftwrap.wrapperstaging's stagein() and stageout() functions. >>>>> That's where you would add additional protocols. >>>>> >>>>> The wrapperStagingLocalServer sets the default staging location for >>>>> things that do not have an explicit scheme. In the case above, "c.txt" >>>>> and "a.txt" are staged in and out using"file://". If you wanted the >>>>> default to be some gridftp server, you would say something like >>>>> 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course, >>>>> _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs, >>>>> which it doesn't at the moment. >>>>> >>>>> Mihael -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wilde at anl.gov Sun Nov 9 14:33:15 2014 From: wilde at anl.gov (Michael Wilde) Date: Sun, 9 Nov 2014 14:33:15 -0600 Subject: [Swift-devel] Problems Re: Example of wrapper staging? In-Reply-To: <545FCDA9.9030007@anl.gov> References: <545E5E1D.8000702@anl.gov> <1415484807.975.10.camel@echo> <545E9B2A.1020309@anl.gov> <1415486509.1864.2.camel@echo> <545EA815.7020602@anl.gov> <545FC8F5.5000007@anl.gov> <545FCDA9.9030007@anl.gov> Message-ID: <545FCF8B.5060608@anl.gov> As far as I can tell, when "sh" is the executable, the actual output file produced in the job dir is incorrect. From what I see so far, it looks like _swiftwrap.wrapperstaging may be forming a command line that doesnt work when the app is a "sh -c". Still digging. - Mike On 11/9/14 2:25 PM, Michael Wilde wrote: > Below is a simpler example of the failing case: > > $ hostname > midway001 > > # Works with echo as app and wrapperStaging: > > $ cat t03.swift > type file; > > app (file o) echo () { > echo "hello" stdout=filename(o); > } > > file o <"out.txt">; > > o = echo(); > $ cat swift.conf > wrapperStagingLocalServer: "file://" > > app.sh { executable: /bin/sh, maxWallTime: "00:01:00" } > > site.local { > staging: "wrapper" > scratch: "/tmp/wilde/swiftest" > workDirectory: "/tmp/wilde/swiftwork" > } > > lazyErrors: false > executionRetries: 0 > keepSiteDir: true > alwaysTransferWrapperLog: true > > > $ rm out.txt > > # Fails with sh as app and wrapper staging: > > $ swift -config swift.conf t03.swift > Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 > heads/master 6172 > RunID: run011 > Progress: Sun, 09 Nov 2014 20:20:39+0000 > Final status:Sun, 09 Nov 2014 20:20:40+0000 Finished successfully:1 > $ cat out.txt > hello > $ cat t04.swift > type file; > > app (file o) echo () { > sh "-c" "/bin/echo hello" stdout=filename(o); > } > > file o <"out.txt">; > > o = echo(); > $ swift -config swift.conf t04.swift > Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 > heads/master 6172 > RunID: run012 > Progress: Sun, 09 Nov 2014 20:21:09+0000 > Final status:Sun, 09 Nov 2014 20:21:09+0000 Finished successfully:1 > $ cat out.txt > > $ ls -l out.txt > -rw-rw-r-- 1 wilde wilde 1 Nov 9 14:21 out.txt > $ od -x out.txt > 0000000 000a > 0000001 > > # Works with sh as app and default staging: > > $ swift -config swift.conf.nows t04.swift > Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 > heads/master 6172 > RunID: run013 > Progress: Sun, 09 Nov 2014 20:21:59+0000 > Final status:Sun, 09 Nov 2014 20:21:59+0000 Finished successfully:1 > $ cat swift.conf.nows > wrapperStagingLocalServer: "file://" > > app.sh { executable: /bin/sh, maxWallTime: "00:01:00" } > > lazyErrors: false > executionRetries: 0 > > $ cat out.txt > hello > $ > > On 11/9/14 2:05 PM, Michael Wilde wrote: >> Oddly, when I use "sh" -> /bin/sh for the app, this example fails. The >> swift run succeeds, but the returned file is zero-length (for the >> example below) or contains a single newline (when I test a sh -c of a >> simple echo). >> >> It *seems* that using cat as the app works while /bin/sh fails to return >> output correctly. >> >> I'll need to debug further, but below are the working and then failing >> swift scripts: >> >> # This works, c.txt comes back as 1M+12 bytes: >> >> $ cat wrapstage.swift >> type file; >> >> app (file c) cat(file a, file b) { >> cat filename(a) filename(b) stdout=filename(c); >> } >> >> file c <"c.txt">; >> file a <"a.txt">; >> file b <"http://mirror.internode.on.net/pub/test/1meg.test">; >> >> c = cat(a, b); >> >> # this fails: c.txt comes back as zero length: >> >> $ cat wrapstage2.swift >> type file; >> >> app (file c) cat(file a, file b) { >> sh "-c" strcat("/bin/cat ", filename(a), " ", filename(b)) >> stdout=filename(c); >> } >> >> file c <"c.txt">; >> file a <"a.txt">; >> file b <"http://mirror.internode.on.net/pub/test/1meg.test">; >> >> c = cat(a, b); >> swift$ >> >> A bug, or user error? >> >> - Mike >> >> On 11/8/14 5:32 PM, Michael Wilde wrote: >>> Ok, that works great now, both with the conf you posted and with this one: >>> >>> wrapperStagingLocalServer: "file://" >>> >>> site.local { >>> staging: "wrapper" >>> scratch: "/tmp/wilde/swiftest" >>> } >>> >>> Thanks, >>> >>> - Mike >>> >>> On 11/8/14 4:41 PM, Mihael Hategan wrote: >>>> (solved on IM but replying here for reference). >>>> >>>> I made a mistake in the swift.conf I sent. wrapperStagingLocalServer is >>>> a global option not a site option. However, it should work with the >>>> vanilla swift.conf posted, which had the wrong option commented out. >>>> >>>> Mihael >>>> >>>> >>>> On Sat, 2014-11-08 at 16:37 -0600, Michael Wilde wrote: >>>>> I get this error: >>>>> >>>>> org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected >>>>> property 'site.local.wrapperStagingLocalServer' >>>>> at >>>>> org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214) >>>>> >>>>> Ive tried spelling this property both as in your sample .conf and as in >>>>> your text: >>>>> >>>>> wrapperStagingLocalServer and wrapperStageingLocalServer >>>>> >>>>> Neither is accepted. Did a schema change not get committed? >>>>> >>>>> For general info: where is the json schema file that a user can check to >>>>> find all valid conf parameters? >>>>> >>>>> Thanks, >>>>> >>>>> - Mike >>>>> >>>>> >>>>> >>>>> On 11/8/14 4:13 PM, Mihael Hategan wrote: >>>>>> type file; >>>>>> >>>>>> app (file c) cat(file a, file b) { >>>>>> cat filename(a) filename(b) stdout=filename(c); >>>>>> } >>>>>> >>>>>> file c <"c.txt">; >>>>>> file a <"a.txt">; >>>>>> file b <"http://mirror.internode.on.net/pub/test/1meg.test">; >>>>>> >>>>>> c = cat(a, b); >>>>>> -------------------------------------------------------- >>>>>> >>>>>> You would need a proper swift.conf. Here's an example of that: >>>>>> -------------------------------------------------------- >>>>>> site.local { >>>>>> staging: "wrapper" >>>>>> scratch: "/some/scratch/dir" >>>>>> # *see email comments for details on the following option >>>>>> # wrapperStagingLocalServer:"file://" >>>>>> } >>>>>> -------------------------------------------------------- >>>>>> >>>>>> >>>>>> With a proper scratch dir there, this*should* work for you without any >>>>>> additional tweaking. >>>>>> >>>>>> Check _swiftwrap.wrapperstaging's stagein() and stageout() functions. >>>>>> That's where you would add additional protocols. >>>>>> >>>>>> The wrapperStagingLocalServer sets the default staging location for >>>>>> things that do not have an explicit scheme. In the case above, "c.txt" >>>>>> and "a.txt" are staged in and out using"file://". If you wanted the >>>>>> default to be some gridftp server, you would say something like >>>>>> 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course, >>>>>> _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs, >>>>>> which it doesn't at the moment. >>>>>> >>>>>> Mihael -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wilde at anl.gov Mon Nov 10 17:25:31 2014 From: wilde at anl.gov (Michael Wilde) Date: Mon, 10 Nov 2014 17:25:31 -0600 Subject: [Swift-devel] Example of wrapper staging? In-Reply-To: <1415484807.975.10.camel@echo> References: <545E5E1D.8000702@anl.gov> <1415484807.975.10.camel@echo> Message-ID: <5461496B.7030106@anl.gov> Mihael, can you clarify: is the reason that a scratch tag is required simply that the stagein and stageout functions in _swiftwrap.wrapperstaging are coded to expect it? But that limitation could eventually be relaxed? - Mike On 11/8/14 4:13 PM, Mihael Hategan wrote: > With a proper scratch dir there, this*should* work for you without any > additional tweaking. -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Tue Nov 11 16:22:02 2014 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Tue, 11 Nov 2014 16:22:02 -0600 Subject: [Swift-devel] CLUSTER 2015 - Call for Workshops Message-ID: <54628C0A.2060607@cs.iit.edu> IEEE International Conference on Cluster Computing September 8-11, 2015 Chicago, IL, USA https://press3.mcs.anl.gov/ieeecluster2015/ ---------------------------------------------- ...Follow us on Facebook athttps://www.facebook.com/ieee.cluster ...Follow us on Twitter athttps://twitter.com/IEEECluster ...Follow us on Linkedin at https://www.linkedin.com/groups/IEEE-International-Conference-on-Cluster-7428925 ...Follow us on RenRen athttp://page.renren.com/601871401 ---------------------------------------------- CALL FOR WORKSHOPS Experts in cluster computing are invited to share their expertise with the community by submitting proposals for workshops to be co-located with the Cluster 2015 conference that will be held on September 8-11, 2015 in Chicago, Illinois. The Cluster 2015 workshop program will provide an opportunity for authors to present preliminary work related to the topics covered by Cluster 2015 and, thus, will give attendees access to the latest work in progress in the field. Cluster 2015 invites proposals for workshops on a broad range of topics on the design of clusters and related technologies and on effectively managing or using cluster resources and services. All workshops will be held on September 8, 2015. TOPIC CATEGORIES Topic Categories of interest include but are not limited to the following: Cluster node and system hardware; Cluster/distributed operating and runtime systems; Parallel programming models, language, runtimes and environments; Performance evaluation, analysis and optimization; Fault-tolerance, resilience, reliability and high availability; Data management and distributed and parallel file systems; Visualization and data analytics; Cluster management tools including resource management; Power management, energy and cooling; High performance networks and communication systems; Clouds and distributed systems; Security. SUBMISSION GUIDELINES Proposals are limited to a 3 page description that should include the following: A title, an abstract (400 words maximum),and a syllabus; Discussion of the topic's relevance and workshop goals; A description of the target audience; An indication of half-day or full-day duration; If applicable, details on previous instantiations of the workshop, including attendance numbers. In addition to the 3 page description, please include the following: A brief biographical sketch of each workshop organizer (2 pages per organizer). Workshop proposal will be evaluated as they are received. Decisions will be made based on the proposal content and its relationship to any workshop proposals that have been accepted or received at the time the proposal is received. Since this policy may result in a the workshop program being completed early, Cluster 2015 reserves the right to close workshop submissions prior to the date listed below. A single pdf file with all the material described above should be submitted to the Cluster 2015 conference on-line submission system under the workshop track (https://easychair.org/conferences/?conf=cluster2015workshops). Workshop length will be determined based on the following policy: Workshops that accept ten or more papers will be provided space for a full day program; Workshops that accept seven to nine papers will be provided space for four and a half hours (not including breaks); Workshops that accept five or six papers will be provided space for three hours (not including breaks); Workshops that accept three or four papers will be provided space for a half hour per paper; Workshops with fewer accepted papers will be canceled. Cluster 2015 will consider increasing the time allocated to a workshop if space is available. Cluster 2015 will provide free conference registration for one keynote speaker to workshops that accept ten or more papers. The free registration cannot be used by a workshop organizer, even if the organizer presents a keynote. IMPORTANT DATES Workshop proposals are selected and notifications are sent on a first-come basis. Three workshops have already been selected! CLUSTER 2015 WORKSHOP AND TUTORIAL CHAIR Bronis R. de Supinski, Lawrence Livermore National Laboratory (first name at llnl dot gov). -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Tue Nov 11 17:45:47 2014 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Tue, 11 Nov 2014 17:45:47 -0600 Subject: [Swift-devel] Call for Participation: ACM MTAGS 2014 @ SC14 Message-ID: <54629FAB.2000005@cs.iit.edu> 7th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) 2014 *Co-located withSupercomputing/SC 2014 In cooperation with ACM SIGHPC New Orleans, Louisiana -- November 16th, 2014* Highlights * *Keynote speakers:* o Dr. Owen O'Malley , Co-founder Hortonworks Inc. + "/*Where is Hadoop Going Next?*/*"* o Michael Wilde , Fellow at University of Chicago and Software Architect at Argonne National Laboratory + /*"Dataflow approaches to many-task computing: past, present and future"* / * *Panel: Many-Task Computing in a Big Data World* * *Papers:* o /"BoscoR: Extending R from the desktop to the Grid "/, Derek Weitzel, Jaime Frey, Marco Mambelli, Dan Fraser, Miha Ahronovitz, David Swanson o /"Energy Prediction for I/O Intensive Workflow Applications "/, Hao Yang, Lauro Costa, Matei Ripeanu o /"Overhead-Aware-Best-Fit (OABF) Resource Allocation Algorithm for Minimizing VM Launching Overhead "/, Hao Wu, Shangping Ren, Steven Timm, Gabriele Garzoglio, Seo-Young Noh o /"A Many-Task Parallel Approach for Multiscale Simulations of Subsurface Flow and Reactive Transport "/, Timothy Scheibe, Xiaofan Yang, Karen Schuchardt, Jared Chase, Bruce Palmer, Alexandre Tartakovsky o /"Lightweight Superscalar Task Execution in Distributed Memory "/, Asim YarKhan, Jack Dongarra For the full program, see http://datasys.cs.iit.edu/events/MTAGS14/program.html. -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Tue Nov 11 23:08:40 2014 From: wilde at anl.gov (Michael Wilde) Date: Tue, 11 Nov 2014 23:08:40 -0600 Subject: [Swift-devel] How to site-wide environment vars? Message-ID: <5462EB58.9050504@anl.gov> I'm trying (with trunk) to set site-wide env vars that will be seen by _swiftwrap.wrapperstaging (e.g. to indicated the userid to use to connect to globus.org). Is there any way to do this, similar to what's there for setting env vars for specific apps? - Mike -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From hategan at mcs.anl.gov Wed Nov 12 14:39:39 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 12 Nov 2014 12:39:39 -0800 Subject: [Swift-devel] How to site-wide environment vars? In-Reply-To: <5462EB58.9050504@anl.gov> References: <5462EB58.9050504@anl.gov> Message-ID: <1415824779.32590.0.camel@echo> I think app.ALL {env.XYZ:...} should work. Environment variables are set before _swiftwrap is invoked. Mihael On Tue, 2014-11-11 at 23:08 -0600, Michael Wilde wrote: > I'm trying (with trunk) to set site-wide env vars that will be seen by > _swiftwrap.wrapperstaging (e.g. to indicated the userid to use to > connect to globus.org). > > Is there any way to do this, similar to what's there for setting env > vars for specific apps? > > - Mike > From wilde at anl.gov Thu Nov 13 12:03:13 2014 From: wilde at anl.gov (Michael Wilde) Date: Thu, 13 Nov 2014 12:03:13 -0600 Subject: [Swift-devel] minor error from coasters Message-ID: <5464F261.7010803@anl.gov> Running with the latest 0.95 rev Im seeing this error in the worker.pl stderr: Use of uninitialized value $s in concatenation (.) or string at /home/wilde/.globus/coasters/cscript7278897633798109681.pl line 2220. I think there's a ticket on this. Is it fixed in trunk? Its minor noise but noise nonetheless. Mihael or Yadu can you make sure its fixed or that a ticket is still tracking it? Thanks, - Mike -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago