From benc at hawaga.org.uk Tue Nov 11 15:01:52 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 11 Nov 2008 21:01:52 +0000 (GMT) Subject: [Swift-user] Swift 0.7 released. Message-ID: Swift v0.7 is now available for download from the Swift website, at http://www.ci.uchicago.edu/swift/downloads/index.php Release notes for this release can be found at http://www.ci.uchicago.edu/swift/downloads/release-notes-0.7.txt -- From iraicu at cs.uchicago.edu Fri Nov 14 16:34:03 2008 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Fri, 14 Nov 2008 16:34:03 -0600 Subject: [Swift-user] Falkon related activities at SC08 Message-ID: <491DFCDB.8040706@cs.uchicago.edu> Hi all, I setup a wiki (http://dev.globus.org/wiki/SC08_Activities) outlining the various talks or events that are taking place next week at SC08 that might be of interest. Hope to see some of you there! Cheers, Ioan -- =================================================== Ioan Raicu Ph.D. Candidate =================================================== Distributed Systems Laboratory Computer Science Department University of Chicago 1100 E. 58th Street, Ryerson Hall Chicago, IL 60637 =================================================== Email: iraicu at cs.uchicago.edu Web: http://www.cs.uchicago.edu/~iraicu http://dev.globus.org/wiki/Incubator/Falkon http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page =================================================== =================================================== From wilde at mcs.anl.gov Sat Nov 15 19:34:10 2008 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sat, 15 Nov 2008 19:34:10 -0600 Subject: [Swift-user] backslash quoting in strings Message-ID: <491F7892.20200@mcs.anl.gov> It seems from experimentation that backslash quoting changed at some point (or the user guide and tutorial are wrong): the \1 needed in the transform arg of the regexp_mapper needs to be \\1 - at least it seems this way from experimentation with trace(). Its also odd trace() takes \t and \n but echoes them just like that, with the backslashes printing. Are these the intended behaviors? From benc at hawaga.org.uk Sat Nov 15 19:42:28 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Sun, 16 Nov 2008 01:42:28 +0000 (GMT) Subject: [Swift-user] backslash quoting in strings In-Reply-To: <491F7892.20200@mcs.anl.gov> References: <491F7892.20200@mcs.anl.gov> Message-ID: On Sat, 15 Nov 2008, Michael Wilde wrote: > It seems from experimentation that backslash quoting changed at some point (or > the user guide and tutorial are wrong): the \1 needed in the transform arg of > the regexp_mapper needs to be \\1 - at least it seems this way from > experimentation with trace(). Yes, that changed between v0.4 and v0.5. This is an intended behaviour, to facilitate putting in \" characters. > Its also odd trace() takes \t and \n but echoes them just like that, with the > backslashes printing. I don't think has had any strong intention about that should behave. On looking at the parser, \ followed by one of n r t b f " \ will be passed through, with apparently only the quotes having some special behaviour and the others passing through unmodified. Other \ sequences, such as \g, will cause a parser error. This could be tidied up sometime. File a bug for it. -- From wilde at mcs.anl.gov Sat Nov 15 20:50:10 2008 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sat, 15 Nov 2008 20:50:10 -0600 Subject: [Swift-user] suggestion for program flow control In-Reply-To: References: <380944.70847.qm@web52304.mail.re2.yahoo.com> Message-ID: <491F8A62.4090702@mcs.anl.gov> Can this technique be readily applied to a function that runs N app() functions in a foreach()? (external o) doall(Collection c) { foreach i in c { x = someapp(); } } app sync(external o) {etc} sync=doall() summarize(sync); Any way to make the completion of doall() dependent on the completion of all the someapp() calls? - Mike On 7/8/08 1:59 AM, Ben Clifford wrote: > On Tue, 17 Jun 2008, Ben Clifford wrote: > >>> I can definitely see the benefit of having separate pipelines for >>> non-dependent parts within the same script, but perhaps there is a way >>> to chain dependent functions that is not dependent on files produced by >>> previous functions? >> I've been playing with some code to do that as someone else requested it. >> >> Basically you will be able to have a swiftscript variable that expresses >> the dependency, but doesn't have any actual content (such as a file). >> >> Hopefully later this week there will be something in SVN. > > Somewhat later than I'd hoped. Swift SVN r2095 has 'extern' types. You can > use like this: > > > (external o) a() { > app { > helperA @strcat(@arg("dir"),"/restart-extern.1.out") "/etc/group" > "qux"; > } > } > > b(external o) { > app { > helperB @strcat(@arg("dir"),"/restart-extern.2.out") "/etc/group" > "baz"; > } > } > > > external sync; > > sync=a(); > b(sync); > > This makes a dependency between a and b, but doesn't actually move any > data around; its entirely up to you to ensure that when the a procedure > finishes your data is in the right place for b to find it. > From benc at hawaga.org.uk Sat Nov 15 21:07:09 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Sun, 16 Nov 2008 03:07:09 +0000 (GMT) Subject: [Swift-user] suggestion for program flow control In-Reply-To: <491F8A62.4090702@mcs.anl.gov> References: <380944.70847.qm@web52304.mail.re2.yahoo.com> <491F8A62.4090702@mcs.anl.gov> Message-ID: On Sat, 15 Nov 2008, Michael Wilde wrote: > Any way to make the completion of doall() dependent on the completion of all > the someapp() calls? Try this. Represent the output of each separate call with a separate 'external'. Store them in an array. When the array is closed, things that depend on that array can complete. The code that you specify doesn't work because you can only use externals as return values from application procedures, not from compound procedures. (the semantics of returning from compound procedures are poorly defined - I haven't thought about them much) app (external o) a(int num) { echo "first" num stdout="/dev/tty" ; } (external o[]) first() { foreach i in [1:10] { o[i]=a(i); } } app b(external o[]) { echo "second" stdout="/dev/tty"; } external sync[] = first(); b(sync); -- From wilde at mcs.anl.gov Sun Nov 16 11:25:45 2008 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sun, 16 Nov 2008 11:25:45 -0600 Subject: [Swift-user] Running swift from firewalled host Message-ID: <49205799.1020508@mcs.anl.gov> Will swift running on a host with limited inbound port connectivity be able to run jobs on a WS-GRAM site? Based on the Globus firewall wiki page seems it should work (see below), but thought I would ask before trying it. - Mike WS-GRAM: Connections back to client (controllable ephemeral port to controllable ephemeral port) required if executable or data staged from client or output from job sent back to client. ^^^ I assume Swift does not do either of these? Port 8443/tcp is default and configurable. GirdFTP: In the case of a single data channel, the connection is established from a controllable ephemeral port on the client to a controllable ephemeral port on the server. ^^^ Is this is what Swift does? From hategan at mcs.anl.gov Sun Nov 16 11:39:23 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 16 Nov 2008 11:39:23 -0600 Subject: [Swift-user] Running swift from firewalled host In-Reply-To: <49205799.1020508@mcs.anl.gov> References: <49205799.1020508@mcs.anl.gov> Message-ID: <1226857163.12232.8.camel@localhost> On Sun, 2008-11-16 at 11:25 -0600, Michael Wilde wrote: > Will swift running on a host with limited inbound port connectivity be > able to run jobs on a WS-GRAM site? > > Based on the Globus firewall wiki page Link? > seems it should work (see below), > but thought I would ask before trying it. > > - Mike > > WS-GRAM: > > Connections back to client (controllable ephemeral port to controllable > ephemeral port) required if executable or data staged from client or > output from job sent back to client. > > ^^^ I assume Swift does not do either of these? Yes, however what that text fails to mention is that only polling for job status will work (not notifications). Swift does not use polling, so you need a an open port for the notification server. > > Port 8443/tcp is default and configurable. > > GirdFTP: > > In the case of a single data channel, the connection is established from > a controllable ephemeral port on the client to a controllable ephemeral > port on the server. > > ^^^ Is this is what Swift does? Yes. So that's one additional port for each concurrent transfer. Basically you need a reasonable port range. From wilde at mcs.anl.gov Sun Nov 16 12:29:38 2008 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sun, 16 Nov 2008 12:29:38 -0600 Subject: [Swift-user] Running swift from firewalled host In-Reply-To: <1226857163.12232.8.camel@localhost> References: <49205799.1020508@mcs.anl.gov> <1226857163.12232.8.camel@localhost> Message-ID: <49206692.1090303@mcs.anl.gov> On 11/16/08 11:39 AM, Mihael Hategan wrote: > On Sun, 2008-11-16 at 11:25 -0600, Michael Wilde wrote: >> Will swift running on a host with limited inbound port connectivity be >> able to run jobs on a WS-GRAM site? >> >> Based on the Globus firewall wiki page > > Link? http://dev.globus.org/wiki/FirewallHowTo >> seems it should work (see below), >> but thought I would ask before trying it. >> >> - Mike >> >> WS-GRAM: >> >> Connections back to client (controllable ephemeral port to controllable >> ephemeral port) required if executable or data staged from client or >> output from job sent back to client. >> >> ^^^ I assume Swift does not do either of these? > > Yes, however what that text fails to mention is that only polling for > job status will work (not notifications). Swift does not use polling, so > you need a an open port for the notification server. Alas. So no way to run Swift-to-Globus from a host that you cant open up "controllable" ephemeral ports (like GLOBUS_TCP_PORT_RANGE) on, it seems? - Mike >> Port 8443/tcp is default and configurable. >> >> GirdFTP: >> >> In the case of a single data channel, the connection is established from >> a controllable ephemeral port on the client to a controllable ephemeral >> port on the server. >> >> ^^^ Is this is what Swift does? > > Yes. So that's one additional port for each concurrent transfer. > Basically you need a reasonable port range. > > From hategan at mcs.anl.gov Sun Nov 16 13:15:48 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 16 Nov 2008 13:15:48 -0600 Subject: [Swift-user] Running swift from firewalled host In-Reply-To: <49206692.1090303@mcs.anl.gov> References: <49205799.1020508@mcs.anl.gov> <1226857163.12232.8.camel@localhost> <49206692.1090303@mcs.anl.gov> Message-ID: <1226862948.15604.0.camel@localhost> On Sun, 2008-11-16 at 12:29 -0600, Michael Wilde wrote: > On 11/16/08 11:39 AM, Mihael Hategan wrote: > > On Sun, 2008-11-16 at 11:25 -0600, Michael Wilde wrote: > >> Will swift running on a host with limited inbound port connectivity be > >> able to run jobs on a WS-GRAM site? > >> > >> Based on the Globus firewall wiki page > > > > Link? > > http://dev.globus.org/wiki/FirewallHowTo > > >> seems it should work (see below), > >> but thought I would ask before trying it. > >> > >> - Mike > >> > >> WS-GRAM: > >> > >> Connections back to client (controllable ephemeral port to controllable > >> ephemeral port) required if executable or data staged from client or > >> output from job sent back to client. > >> > >> ^^^ I assume Swift does not do either of these? > > > > Yes, however what that text fails to mention is that only polling for > > job status will work (not notifications). Swift does not use polling, so > > you need a an open port for the notification server. > > Alas. So no way to run Swift-to-Globus from a host that you cant open up > "controllable" ephemeral ports (like GLOBUS_TCP_PORT_RANGE) on, it seems? Right From tiberius at ci.uchicago.edu Tue Nov 25 10:26:45 2008 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Tue, 25 Nov 2008 10:26:45 -0600 Subject: [Swift-user] Restricting/fixing site scores ? Message-ID: Hi I'm trying to run a workflow, and it suffers from various things, one of which being that the score of the site where the app runs gets too low (i think), as I get these: commitDelayedScore(localhost:-10.000(0.039):0/1 overload: -102309, 0.1 I am getting a lot of errors like these: The cache already contains localhost:sparse-iteration-dependency-v3-20081125-1015-cp7py371/shared/param.0.0 Also my workflow behaves non-deterministically: sometimes it finishes correctly, but some other times it throws the cache error above, and just hangs for a very long time. My guess is that the "hanging" happens either because of some synchronization or livelock issue, OR because of the filesystem latency for creating the outputs (e.g. param.0.0) compared to the notification coming from swift that the output files are created. I'm not sure what to do about my issues, but it seems like eliminating the endless wait because of the low score for the site will help with the debugging. Thank you Tibi -- Tiberiu (Tibi) Stef-Praun, PhD Computational Sciences Researcher Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From benc at hawaga.org.uk Tue Nov 25 18:12:57 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 26 Nov 2008 00:12:57 +0000 (GMT) Subject: [Swift-user] Restricting/fixing site scores ? In-Reply-To: References: Message-ID: On Tue, 25 Nov 2008, Tiberiu Stef-Praun wrote: > I am getting a lot of errors like these: > The cache already contains > localhost:sparse-iteration-dependency-v3-20081125-1015-cp7py371/shared/param.0.0 I've mostly seen that error occur when two different variables are mapped to the same filename. Most simply something like file a <"foo">; file b <"foo">; but less obviously in cases where mappers are used inside loops but not parameterised by the iteration variables of all containing loops (eg. for a in [1:5] { file a <"foo">; ... } maps the file "foo" 5 times to different variables (variable a in 5 different variable scopes, one for each iteration of the foreach). > Also my workflow behaves non-deterministically: sometimes it finishes > correctly, but some other times it throws the cache error above, and > just hangs for a very long time. Sounds strange. Can you post the code here? -- From benc at hawaga.org.uk Tue Nov 25 18:16:54 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 26 Nov 2008 00:16:54 +0000 (GMT) Subject: [Swift-user] Restricting/fixing site scores ? In-Reply-To: References: Message-ID: On Tue, 25 Nov 2008, Tiberiu Stef-Praun wrote: > I'm not sure what to do about my issues, but it seems like eliminating > the endless wait because of the low score for the site will help with > the debugging. Set ) in your sites.xml entry/entries. That should take away the delay when a site has negative score. --