From hategan at mcs.anl.gov Mon Nov 3 15:50:17 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 03 Nov 2008 15:50:17 -0600 Subject: [Swift-devel] large run plot Message-ID: <1225749017.12247.2.camel@localhost> So I tried to use the log tools to plot this rather large log. Various errors happened which ultimately led to the whole thing failing. The beast is /home/hategan/logs/modgenproc-20081103-1350-1ht9bn7d.log @ci. Mihael From benc at hawaga.org.uk Mon Nov 3 17:11:40 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 3 Nov 2008 23:11:40 +0000 (GMT) Subject: [Swift-devel] large run plot In-Reply-To: <1225749017.12247.2.camel@localhost> References: <1225749017.12247.2.camel@localhost> Message-ID: On Mon, 3 Nov 2008, Mihael Hategan wrote: > So I tried to use the log tools to plot this rather large log. Various > errors happened which ultimately led to the whole thing failing. The > beast is /home/hategan/logs/modgenproc-20081103-1350-1ht9bn7d.log @ci. I just ran with a fresh checkout of log-processing r 2318 (last changed r2296) on communicado, and it completes giving this: http://www.ci.uchicago.edu/~benc/tmp/report-modgenproc-20081103-1350-1ht9bn7d/ On the way, there are some errors about head -n needing parameters, which I have not seen before but will investigate; these are non-fatal except that they prevent the generation of certain graphs. Does this correspond with what you saw? If not, please send more detail. -- From hategan at mcs.anl.gov Mon Nov 3 17:48:36 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 03 Nov 2008 17:48:36 -0600 Subject: [Swift-devel] large run plot In-Reply-To: References: <1225749017.12247.2.camel@localhost> Message-ID: <1225756116.13779.5.camel@localhost> On Mon, 2008-11-03 at 23:11 +0000, Ben Clifford wrote: > > On Mon, 3 Nov 2008, Mihael Hategan wrote: > > > So I tried to use the log tools to plot this rather large log. Various > > errors happened which ultimately led to the whole thing failing. The > > beast is /home/hategan/logs/modgenproc-20081103-1350-1ht9bn7d.log @ci. > > > I just ran with a fresh checkout of log-processing r 2318 (last changed > r2296) on communicado, and it completes giving this: > > http://www.ci.uchicago.edu/~benc/tmp/report-modgenproc-20081103-1350-1ht9bn7d/ Well, that's what I wanted. It's a 65000 job CNARI run on Ranger. May be interesting to look at. Mostly because it runs in about 40 minutes, which I think is a fairly good number for both Swift and CNARI. > > > On the way, there are some errors about head -n needing parameters, which > I have not seen before but will investigate; these are non-fatal except > that they prevent the generation of certain graphs. > > Does this correspond with what you saw? Yes, though there were some other errors. I'll have to update to latest code and try again. From benc at hawaga.org.uk Mon Nov 3 17:51:42 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 3 Nov 2008 23:51:42 +0000 (GMT) Subject: [Swift-devel] large run plot In-Reply-To: <1225756116.13779.5.camel@localhost> References: <1225749017.12247.2.camel@localhost> <1225756116.13779.5.camel@localhost> Message-ID: On Mon, 3 Nov 2008, Mihael Hategan wrote: > Yes, though there were some other errors. I'll have to update to latest > code and try again. probably a better idea to force r2318 rather than use head as I'm actively committing stuff in there at the moment. -- From benc at hawaga.org.uk Mon Nov 3 18:05:44 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 4 Nov 2008 00:05:44 +0000 (GMT) Subject: [Swift-devel] cnari run time In-Reply-To: <1225756116.13779.5.camel@localhost> References: <1225749017.12247.2.camel@localhost> <1225756116.13779.5.camel@localhost> Message-ID: On Mon, 3 Nov 2008, Mihael Hategan wrote: > Well, that's what I wanted. It's a 65000 job CNARI run on Ranger. May be > interesting to look at. Mostly because it runs in about 40 minutes, > which I think is a fairly good number for both Swift and CNARI. 40 minutes is quite a nice number for SC08 demo purposes. Swift has a 1h slot in the argonne booth, so there is potential for a live demo that shows an appreciable part of the application happening during the demo time. -- From benc at hawaga.org.uk Mon Nov 3 20:32:31 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 4 Nov 2008 02:32:31 +0000 (GMT) Subject: [Swift-devel] 0.7rc1 + local coasters + communicado Message-ID: I just tried running coasters on communicado submitting locally. Several times in a row its got stuck like this for at least 12 minutes: (with different numbers of completed and active jobs, but around this many...) > Progress: Selecting site:978 Active:4 Finished successfully:18 This is running with tests/sites/coasters/coaster-local.xml with tests-sites/run-site modified to run 066-many.swift The same runs ok on my laptop repeatedly. I haven't dug any further. -- From foster at anl.gov Tue Nov 4 05:43:38 2008 From: foster at anl.gov (Ian Foster) Date: Tue, 4 Nov 2008 05:43:38 -0600 Subject: [Swift-devel] large run plot In-Reply-To: <1225756116.13779.5.camel@localhost> References: <1225749017.12247.2.camel@localhost> <1225756116.13779.5.camel@localhost> Message-ID: <6DD6CF16-787E-47B9-8184-18644A943B22@anl.gov> nice! On Nov 3, 2008, at 5:48 PM, Mihael Hategan wrote: > On Mon, 2008-11-03 at 23:11 +0000, Ben Clifford wrote: >> >> On Mon, 3 Nov 2008, Mihael Hategan wrote: >> >>> So I tried to use the log tools to plot this rather large log. >>> Various >>> errors happened which ultimately led to the whole thing failing. The >>> beast is /home/hategan/logs/modgenproc-20081103-1350-1ht9bn7d.log >>> @ci. >> >> >> I just ran with a fresh checkout of log-processing r 2318 (last >> changed >> r2296) on communicado, and it completes giving this: >> >> http://www.ci.uchicago.edu/~benc/tmp/report-modgenproc-20081103-1350-1ht9bn7d/ > > Well, that's what I wanted. It's a 65000 job CNARI run on Ranger. > May be > interesting to look at. Mostly because it runs in about 40 minutes, > which I think is a fairly good number for both Swift and CNARI. > >> >> >> On the way, there are some errors about head -n needing parameters, >> which >> I have not seen before but will investigate; these are non-fatal >> except >> that they prevent the generation of certain graphs. >> >> Does this correspond with what you saw? > > Yes, though there were some other errors. I'll have to update to > latest > code and try again. > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Tue Nov 4 11:15:10 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 04 Nov 2008 11:15:10 -0600 Subject: [Swift-devel] 0.7rc1 + local coasters + communicado In-Reply-To: References: Message-ID: <1225818910.21899.0.camel@localhost> On Tue, 2008-11-04 at 02:32 +0000, Ben Clifford wrote: > > I just tried running coasters on communicado submitting locally. Several > times in a row its got stuck like this for at least 12 minutes: (with > different numbers of completed and active jobs, but around this many...) > > > Progress: Selecting site:978 Active:4 Finished successfully:18 > > This is running with tests/sites/coasters/coaster-local.xml with > tests-sites/run-site modified to run 066-many.swift Heh. Worker stuck trying to write to stderr. > > The same runs ok on my laptop repeatedly. > > I haven't dug any further. From hategan at mcs.anl.gov Tue Nov 4 11:19:15 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 04 Nov 2008 11:19:15 -0600 Subject: [Swift-devel] 0.7rc1 + local coasters + communicado In-Reply-To: <1225818910.21899.0.camel@localhost> References: <1225818910.21899.0.camel@localhost> Message-ID: <1225819155.21966.2.camel@localhost> On Tue, 2008-11-04 at 11:15 -0600, Mihael Hategan wrote: > On Tue, 2008-11-04 at 02:32 +0000, Ben Clifford wrote: > > > > I just tried running coasters on communicado submitting locally. Several > > times in a row its got stuck like this for at least 12 minutes: (with > > different numbers of completed and active jobs, but around this many...) > > > > > Progress: Selecting site:978 Active:4 Finished successfully:18 > > > > This is running with tests/sites/coasters/coaster-local.xml with > > tests-sites/run-site modified to run 066-many.swift > > Heh. Worker stuck trying to write to stderr. > And that's because the local provider first consumes stdout and only then stderr, which means stderr is only dealt with after the process completes. Not that there isn't a comment there saying precisely that. > > > > The same runs ok on my laptop repeatedly. I suppose the difference is in process io buffer sizes. From iraicu at cs.uchicago.edu Tue Nov 4 12:17:50 2008 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 04 Nov 2008 12:17:50 -0600 Subject: [Swift-devel] 0.7rc1 + local coasters + communicado In-Reply-To: <1225819155.21966.2.camel@localhost> References: <1225818910.21899.0.camel@localhost> <1225819155.21966.2.camel@localhost> Message-ID: <491091CE.8000301@cs.uchicago.edu> The way I addressed this in Falkon is to have two asynchronous threads that reads both the stdout and stderr at the same time, and maybe a third thread that is waiting for the application thread to terminate, and it works quite well, other than the fact that it uses 4 threads instead of 1 thread. I had a similar issue as you described below, which prompted me to implement any application thread that the worker invokes, to launch these other extra threads to manage the app thread. Ioan Mihael Hategan wrote: > On Tue, 2008-11-04 at 11:15 -0600, Mihael Hategan wrote: > >> On Tue, 2008-11-04 at 02:32 +0000, Ben Clifford wrote: >> >>> I just tried running coasters on communicado submitting locally. Several >>> times in a row its got stuck like this for at least 12 minutes: (with >>> different numbers of completed and active jobs, but around this many...) >>> >>> >>>> Progress: Selecting site:978 Active:4 Finished successfully:18 >>>> >>> This is running with tests/sites/coasters/coaster-local.xml with >>> tests-sites/run-site modified to run 066-many.swift >>> >> Heh. Worker stuck trying to write to stderr. >> >> > > And that's because the local provider first consumes stdout and only > then stderr, which means stderr is only dealt with after the process > completes. Not that there isn't a comment there saying precisely that. > > >>> The same runs ok on my laptop repeatedly. >>> > > I suppose the difference is in process io buffer sizes. > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > -- =================================================== Ioan Raicu Ph.D. Candidate =================================================== Distributed Systems Laboratory Computer Science Department University of Chicago 1100 E. 58th Street, Ryerson Hall Chicago, IL 60637 =================================================== Email: iraicu at cs.uchicago.edu Web: http://www.cs.uchicago.edu/~iraicu http://dev.globus.org/wiki/Incubator/Falkon http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page =================================================== =================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Nov 4 13:45:07 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 04 Nov 2008 13:45:07 -0600 Subject: [Swift-devel] 0.7rc1 + local coasters + communicado In-Reply-To: <1225819155.21966.2.camel@localhost> References: <1225818910.21899.0.camel@localhost> <1225819155.21966.2.camel@localhost> Message-ID: <1225827907.25871.0.camel@localhost> On Tue, 2008-11-04 at 11:19 -0600, Mihael Hategan wrote: > On Tue, 2008-11-04 at 11:15 -0600, Mihael Hategan wrote: > > On Tue, 2008-11-04 at 02:32 +0000, Ben Clifford wrote: > > > > > > I just tried running coasters on communicado submitting locally. Several > > > times in a row its got stuck like this for at least 12 minutes: (with > > > different numbers of completed and active jobs, but around this many...) > > > > > > > Progress: Selecting site:978 Active:4 Finished successfully:18 > > > > > > This is running with tests/sites/coasters/coaster-local.xml with > > > tests-sites/run-site modified to run 066-many.swift > > > > Heh. Worker stuck trying to write to stderr. > > > > And that's because the local provider first consumes stdout and only > then stderr, which means stderr is only dealt with after the process > completes. Not that there isn't a comment there saying precisely that. Should be fixed in cog r2257. From benc at hawaga.org.uk Wed Nov 5 13:58:25 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 5 Nov 2008 19:58:25 +0000 (GMT) Subject: [Swift-devel] 0.7rc1 + local coasters + communicado In-Reply-To: <1225827907.25871.0.camel@localhost> References: <1225818910.21899.0.camel@localhost> <1225819155.21966.2.camel@localhost> <1225827907.25871.0.camel@localhost> Message-ID: On Tue, 4 Nov 2008, Mihael Hategan wrote: > Should be fixed in cog r2257. Seems to cause tests/language-behaviour/001-echo to hang on my laptop, though it worked fine on the NMI build/test per-commit build. -- From benc at hawaga.org.uk Wed Nov 5 14:36:49 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 5 Nov 2008 20:36:49 +0000 (GMT) Subject: [Swift-devel] 0.7rc1 + local coasters + communicado In-Reply-To: References: <1225818910.21899.0.camel@localhost> <1225819155.21966.2.camel@localhost> <1225827907.25871.0.camel@localhost> Message-ID: The below one-liner seems to fix hanging on my laptop. I didn't commit in case I'm missing something in the logic, but I think its ok: --- src/org/globus/cog/abstraction/impl/execution/local/JobSubmissionTaskHandler.java (revision 2257) +++ src/org/globus/cog/abstraction/impl/execution/local/JobSubmissionTaskHandler.java (working copy) @@ -342,6 +342,7 @@ boolean any = processPairs(); if (processDone()) { closePairs(); + return; } else { if (!any) Also, though I don't see that it would cause a lock, this looks wrong: int avail = sp.is.available(); if (avail > 0) { any = true; int len = sp.is.read(); sp.os.write(buf, 0, len); } InputStream.read() with no parameter returns the read byte, not a buffer length. read(buf, offset, len) is perhaps what is intended - I think at present its going to send out big wads of empty space instead of stream content. Also, I was wrong about it passing on the nmi buid and test system - that test is still queued to run at the moment. -- From hategan at mcs.anl.gov Wed Nov 5 14:55:21 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 05 Nov 2008 14:55:21 -0600 Subject: [Swift-devel] 0.7rc1 + local coasters + communicado In-Reply-To: References: <1225818910.21899.0.camel@localhost> <1225819155.21966.2.camel@localhost> <1225827907.25871.0.camel@localhost> Message-ID: <1225918522.20614.0.camel@localhost> On Wed, 2008-11-05 at 19:58 +0000, Ben Clifford wrote: > On Tue, 4 Nov 2008, Mihael Hategan wrote: > > > Should be fixed in cog r2257. > > Seems to cause tests/language-behaviour/001-echo to hang on my laptop, > though it worked fine on the NMI build/test per-commit build. > I'm not that surprised. Any details? From hategan at mcs.anl.gov Wed Nov 5 15:00:28 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 05 Nov 2008 15:00:28 -0600 Subject: [Swift-devel] 0.7rc1 + local coasters + communicado In-Reply-To: References: <1225818910.21899.0.camel@localhost> <1225819155.21966.2.camel@localhost> <1225827907.25871.0.camel@localhost> Message-ID: <1225918828.20614.4.camel@localhost> On Wed, 2008-11-05 at 20:36 +0000, Ben Clifford wrote: > The below one-liner seems to fix hanging on my laptop. I didn't commit in > case I'm missing something in the logic, but I think its ok: > > > --- > src/org/globus/cog/abstraction/impl/execution/local/JobSubmissionTaskHandler.java > (revision 2257) > +++ > src/org/globus/cog/abstraction/impl/execution/local/JobSubmissionTaskHandler.java > (working copy) > @@ -342,6 +342,7 @@ > boolean any = processPairs(); > if (processDone()) { > closePairs(); > + return; Right. That's how it's supposed to be. > } > else { > if (!any) > > > > Also, though I don't see that it would cause a lock, this looks wrong: > > int avail = sp.is.available(); > if (avail > 0) { > any = true; > int len = sp.is.read(); > sp.os.write(buf, 0, len); > } > Also right. It's supposed to be sp.is.read(buf). Fixed these in r2259. > > InputStream.read() with no parameter returns the read byte, not a buffer > length. read(buf, offset, len) is perhaps what is intended - I think at > present its going to send out big wads of empty space instead of stream > content. > > Also, I was wrong about it passing on the nmi buid and test system - that > test is still queued to run at the moment. I was about to be surprised that NMI computers have found a way of gracefully exiting an infinite loop. From zhaozhang at uchicago.edu Thu Nov 6 14:39:38 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 06 Nov 2008 14:39:38 -0600 Subject: [Swift-devel] discussion for swift output file path Message-ID: <4913560A.7050000@uchicago.edu> Hi, All I am working on integrate the Collective IO system and swift on BGP. Before that, for the purpose of put swift into production work, we need to change the output file path. For now, wrapper.sh would copy all output files to jobdir/shared/, on BGP, all output files will be written to one directory, which I am sure will cause the GPFS lock mechanism, thus introduce unacceptable latency. So the easiest way to fix this is "make a hierarchical directory in shared/ and we already did in info/ and jobs/". Several changes we need: place-need-change diffculty 1. change vdl-int.k: create hierarchical directory in shared/, straightforward 2. change wrapper.sh: copy files from local ramdisck to GPFS using dd instead of cp, straghtforward. 3. change somewhere in swift to make swift know where the data is, the path of the output file in jobdir/shared/ unknown Any comments will be appreciated. best wishes zhangzhao From benc at hawaga.org.uk Thu Nov 6 18:51:00 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 7 Nov 2008 00:51:00 +0000 (GMT) Subject: [Swift-devel] discussion for swift output file path In-Reply-To: <4913560A.7050000@uchicago.edu> References: <4913560A.7050000@uchicago.edu> Message-ID: The wrapper does not necessarily put all output files in one directory. It puts them in a directory structure reflecting the submit-side directory structure. For example, if you map a file "a/b", it will end up in the shared directory as shared/a/b This is exposed to the user, though, rather than being hidden as with the other uses of subdirectories. There are a few places throughout the source code where its assumed that the path locally and remote path are basically the same (modulo base directory). I think its probably fairly straightforward to make it use different names, thoughLook for lines in vdl-int.k that look like this: task:transfer(srcprovider=provider, srchost=srchost, srcfile=filename, srcdir=srcdir, desthost=host, destdir=destdir) ) You can put a different destdir in there, based on (for example) some hash of the filename. Simialrly hash the filenames when they are passed as inputs to wrapper.sh in the vdl:execute line in vdl-int.k On Thu, 6 Nov 2008, Zhao Zhang wrote: > Hi, All > > I am working on integrate the Collective IO system and swift on BGP. Before > that, for the purpose of put swift into production work, > we need to change the output file path. For now, wrapper.sh would copy all > output files to jobdir/shared/, on BGP, all output files will > be written to one directory, which I am sure will cause the GPFS lock > mechanism, thus introduce unacceptable latency. > > So the easiest way to fix this is "make a hierarchical directory in shared/ > and we already did in info/ and jobs/". Several changes we need: > place-need-change > diffculty > 1. change vdl-int.k: create hierarchical directory in shared/, > straightforward > 2. change wrapper.sh: copy files from local ramdisck to GPFS using dd instead > of cp, straghtforward. > 3. change somewhere in swift to make swift know where the data is, the path of > the output file in jobdir/shared/ unknown > > Any comments will be appreciated. > > best wishes > zhangzhao > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From hategan at mcs.anl.gov Thu Nov 6 18:59:54 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 06 Nov 2008 18:59:54 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: References: <4913560A.7050000@uchicago.edu> Message-ID: <1226019594.1602.5.camel@localhost> I spoke about this with Zhao a few days ago. Regardless of the local configuration (which is likely to happen on a fast, non-distributed FS), it may make sense to employ, for application/shared data, a similar scheme to that used for info files. It may also make sense to coalesce all the trees into one, such that data, info, and temporary job dirs are created in the same directory (e.g. (info/a/0, job/a/0) becomes (a/0/info, a/0/job)), since it will reduce the overall number of operations. On Fri, 2008-11-07 at 00:51 +0000, Ben Clifford wrote: > The wrapper does not necessarily put all output files in one directory. > > It puts them in a directory structure reflecting the submit-side directory > structure. For example, if you map a file "a/b", it will end up in the > shared directory as shared/a/b > > This is exposed to the user, though, rather than being hidden as with the > other uses of subdirectories. > > There are a few places throughout the source code where its assumed that > the path locally and remote path are basically the same (modulo base > directory). I think its probably fairly straightforward to make it use > different names, thoughLook for lines in vdl-int.k that look like this: > > task:transfer(srcprovider=provider, > srchost=srchost, srcfile=filename, > srcdir=srcdir, > desthost=host, destdir=destdir) > ) > > > You can put a different destdir in there, based on (for example) some hash > of the filename. Simialrly hash the filenames when they are passed as > inputs to wrapper.sh in the vdl:execute line in vdl-int.k > > On Thu, 6 Nov 2008, Zhao Zhang wrote: > > > Hi, All > > > > I am working on integrate the Collective IO system and swift on BGP. Before > > that, for the purpose of put swift into production work, > > we need to change the output file path. For now, wrapper.sh would copy all > > output files to jobdir/shared/, on BGP, all output files will > > be written to one directory, which I am sure will cause the GPFS lock > > mechanism, thus introduce unacceptable latency. > > > > So the easiest way to fix this is "make a hierarchical directory in shared/ > > and we already did in info/ and jobs/". Several changes we need: > > place-need-change > > diffculty > > 1. change vdl-int.k: create hierarchical directory in shared/, > > straightforward > > 2. change wrapper.sh: copy files from local ramdisck to GPFS using dd instead > > of cp, straghtforward. > > 3. change somewhere in swift to make swift know where the data is, the path of > > the output file in jobdir/shared/ unknown > > > > Any comments will be appreciated. > > > > best wishes > > zhangzhao > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From benc at hawaga.org.uk Thu Nov 6 19:07:25 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 7 Nov 2008 01:07:25 +0000 (GMT) Subject: [Swift-devel] discussion for swift output file path In-Reply-To: <1226019594.1602.5.camel@localhost> References: <4913560A.7050000@uchicago.edu> <1226019594.1602.5.camel@localhost> Message-ID: On Thu, 6 Nov 2008, Mihael Hategan wrote: > It may also make sense to coalesce all the trees into one, such that > data, info, and temporary job dirs are created in the same directory > (e.g. (info/a/0, job/a/0) becomes (a/0/info, a/0/job)), since it will > reduce the overall number of operations. Perhaps does for info and job where the access/lock patterns are the same under both the info and job tree. But the shared data tree needs to be hashed by something based on the file (eg a hash of the filename) rather than on something specific to a particular job. Putting the shared directories under the same hash tree as info/ and job/ would reduce the number of directories in total but would increase lock contention on those directories, because cached data file accesses follow a different pattern from info/ and job/ accesses. -- From hategan at mcs.anl.gov Thu Nov 6 19:15:53 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 06 Nov 2008 19:15:53 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: References: <4913560A.7050000@uchicago.edu> <1226019594.1602.5.camel@localhost> Message-ID: <1226020553.2932.5.camel@localhost> On Fri, 2008-11-07 at 01:07 +0000, Ben Clifford wrote: > On Thu, 6 Nov 2008, Mihael Hategan wrote: > > > It may also make sense to coalesce all the trees into one, such that > > data, info, and temporary job dirs are created in the same directory > > (e.g. (info/a/0, job/a/0) becomes (a/0/info, a/0/job)), since it will > > reduce the overall number of operations. > > Perhaps does for info and job where the access/lock patterns are the same > under both the info and job tree. > > But the shared data tree needs to be hashed by something based on the file > (eg a hash of the filename) rather than on something specific to a > particular job. > > Putting the shared directories under the same hash tree as info/ and job/ > would reduce the number of directories in total but would increase lock > contention on those directories, because cached data file accesses follow > a different pattern from info/ and job/ accesses. Only if we assume non-reentrant read locks. And this would need some investigation, but if concurrent reads do not create contention, then there is no problem, because once a job is done, there is no more writing to its directory. From zhaozhang at uchicago.edu Thu Nov 6 19:17:11 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 06 Nov 2008 19:17:11 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: <1226020553.2932.5.camel@localhost> References: <4913560A.7050000@uchicago.edu> <1226019594.1602.5.camel@localhost> <1226020553.2932.5.camel@localhost> Message-ID: <49139717.50700@uchicago.edu> Mihael Hategan wrote: > On Fri, 2008-11-07 at 01:07 +0000, Ben Clifford wrote: > >> On Thu, 6 Nov 2008, Mihael Hategan wrote: >> >> >>> It may also make sense to coalesce all the trees into one, such that >>> data, info, and temporary job dirs are created in the same directory >>> (e.g. (info/a/0, job/a/0) becomes (a/0/info, a/0/job)), since it will >>> reduce the overall number of operations. >>> >> Perhaps does for info and job where the access/lock patterns are the same >> under both the info and job tree. >> >> But the shared data tree needs to be hashed by something based on the file >> (eg a hash of the filename) rather than on something specific to a >> particular job. >> >> Putting the shared directories under the same hash tree as info/ and job/ >> would reduce the number of directories in total but would increase lock >> contention on those directories, because cached data file accesses follow >> a different pattern from info/ and job/ accesses. >> > > Only if we assume non-reentrant read locks. And this would need some > investigation, but if concurrent reads do not create contention, then > there is no problem, because once a job is done, there is no more > writing to its directory. > > well, as far as we know, the concurrent reads do not create contention. zhao From benc at hawaga.org.uk Thu Nov 6 19:23:44 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 7 Nov 2008 01:23:44 +0000 (GMT) Subject: [Swift-devel] discussion for swift output file path In-Reply-To: <1226020553.2932.5.camel@localhost> References: <4913560A.7050000@uchicago.edu> <1226019594.1602.5.camel@localhost> <1226020553.2932.5.camel@localhost> Message-ID: On Thu, 6 Nov 2008, Mihael Hategan wrote: > Only if we assume non-reentrant read locks. And this would need some > investigation, but if concurrent reads do not create contention, then > there is no problem, because once a job is done, there is no more > writing to its directory. If job output files are left in their job directory, rather than put in some hashed-by-filename shared directory, I think that is true. Otherwise, the access patterns for any given hash bucket for a hash-on-filename are going to be fairly random, with reads and writes happening at all sorts of different times and from many different places. Keeping output files in job-specific directories seems a reasonable thing to do performance wise, but means more awareness on the submit side about which files came from which job. -- From hategan at mcs.anl.gov Thu Nov 6 19:28:09 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 06 Nov 2008 19:28:09 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: References: <4913560A.7050000@uchicago.edu> <1226019594.1602.5.camel@localhost> <1226020553.2932.5.camel@localhost> Message-ID: <1226021289.3276.0.camel@localhost> On Fri, 2008-11-07 at 01:23 +0000, Ben Clifford wrote: > On Thu, 6 Nov 2008, Mihael Hategan wrote: > > > Only if we assume non-reentrant read locks. And this would need some > > investigation, but if concurrent reads do not create contention, then > > there is no problem, because once a job is done, there is no more > > writing to its directory. > > If job output files are left in their job directory, rather than put in > some hashed-by-filename shared directory, I think that is true. Otherwise, > the access patterns for any given hash bucket for a hash-on-filename are > going to be fairly random, with reads and writes happening at all sorts of > different times and from many different places. > > Keeping output files in job-specific directories seems a reasonable thing > to do performance wise, but means more awareness on the submit side about > which files came from which job. We already have awareness of which file on which site. I don't think this is any different. From benc at hawaga.org.uk Mon Nov 10 11:19:02 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 10 Nov 2008 17:19:02 +0000 (GMT) Subject: [Swift-devel] swift 0.7rc1 - please test In-Reply-To: References: Message-ID: I guess no one else tested this much? -- From benc at hawaga.org.uk Tue Nov 11 13:51:15 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 11 Nov 2008 19:51:15 +0000 (GMT) Subject: [Swift-devel] deprecations Message-ID: I'd like to deprecate the following over the next few releases: The element in sites.xml - instead, the element should be used for everything that is used for. The old-style p() { app { ... }} syntax, which is replaced in 0.7 by a prefix app keyword style like this: app p() { ... } -- From bugzilla-daemon at mcs.anl.gov Tue Nov 11 14:08:14 2008 From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov) Date: Tue, 11 Nov 2008 14:08:14 -0600 (CST) Subject: [Swift-devel] [Bug 156] New: Identify Swift as generator of gridftp client info Message-ID: http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=156 Summary: Identify Swift as generator of gridftp client info Product: Swift Version: unspecified Platform: Macintosh OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: benc at hawaga.org.uk ReportedBy: benc at hawaga.org.uk GridFTP developers (buzz, mlink) requested that Swift identify itself in gridftp CLIENTINFO, rather than defaulting to the CoG identification that JGlobus (as of 2008-10-29) produces (or nothing at all, given old version of jglobus in cog head at this time). -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You reported the bug, or are watching the reporter. You are the assignee for the bug, or are watching the assignee. From bugzilla-daemon at mcs.anl.gov Tue Nov 11 14:08:14 2008 From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov) Date: Tue, 11 Nov 2008 14:08:14 -0600 (CST) Subject: [Swift-devel] [Bug 157] New: Identify Swift as generator of gridftp client info Message-ID: http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=157 Summary: Identify Swift as generator of gridftp client info Product: Swift Version: unspecified Platform: Macintosh OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: benc at hawaga.org.uk ReportedBy: benc at hawaga.org.uk GridFTP developers (buzz, mlink) requested that Swift identify itself in gridftp CLIENTINFO, rather than defaulting to the CoG identification that JGlobus (as of 2008-10-29) produces (or nothing at all, given old version of jglobus in cog head at this time). -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You reported the bug, or are watching the reporter. You are the assignee for the bug, or are watching the assignee. From benc at hawaga.org.uk Tue Nov 11 15:01:52 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 11 Nov 2008 21:01:52 +0000 (GMT) Subject: [Swift-devel] Swift 0.7 released. Message-ID: Swift v0.7 is now available for download from the Swift website, at http://www.ci.uchicago.edu/swift/downloads/index.php Release notes for this release can be found at http://www.ci.uchicago.edu/swift/downloads/release-notes-0.7.txt -- From benc at hawaga.org.uk Tue Nov 11 15:50:46 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 11 Nov 2008 21:50:46 +0000 (GMT) Subject: [Swift-devel] gram client usage logging Message-ID: GridFTP has a client info string, to indicate which client is being used. GridFTP people suggested that Swift should specify itself, rather than using the default CoG message that will be inherited from our use of the CoG libraries. Is gram looking at doing anything like this, to allow clients to specify their identity? I know its been tossed around as an idea before. -- From zhaozhang at uchicago.edu Thu Nov 13 00:08:28 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 13 Nov 2008 00:08:28 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: References: <4913560A.7050000@uchicago.edu> Message-ID: <491BC45C.4000502@uchicago.edu> Hi, Ben Are you working at CI tomorrow? I mean Thursday. So we could work together to make this working. zhao Ben Clifford wrote: > The wrapper does not necessarily put all output files in one directory. > > It puts them in a directory structure reflecting the submit-side directory > structure. For example, if you map a file "a/b", it will end up in the > shared directory as shared/a/b > > This is exposed to the user, though, rather than being hidden as with the > other uses of subdirectories. > > There are a few places throughout the source code where its assumed that > the path locally and remote path are basically the same (modulo base > directory). I think its probably fairly straightforward to make it use > different names, thoughLook for lines in vdl-int.k that look like this: > > task:transfer(srcprovider=provider, > srchost=srchost, srcfile=filename, > srcdir=srcdir, > desthost=host, destdir=destdir) > ) > > > You can put a different destdir in there, based on (for example) some hash > of the filename. Simialrly hash the filenames when they are passed as > inputs to wrapper.sh in the vdl:execute line in vdl-int.k > > On Thu, 6 Nov 2008, Zhao Zhang wrote: > > >> Hi, All >> >> I am working on integrate the Collective IO system and swift on BGP. Before >> that, for the purpose of put swift into production work, >> we need to change the output file path. For now, wrapper.sh would copy all >> output files to jobdir/shared/, on BGP, all output files will >> be written to one directory, which I am sure will cause the GPFS lock >> mechanism, thus introduce unacceptable latency. >> >> So the easiest way to fix this is "make a hierarchical directory in shared/ >> and we already did in info/ and jobs/". Several changes we need: >> place-need-change >> diffculty >> 1. change vdl-int.k: create hierarchical directory in shared/, >> straightforward >> 2. change wrapper.sh: copy files from local ramdisck to GPFS using dd instead >> of cp, straghtforward. >> 3. change somewhere in swift to make swift know where the data is, the path of >> the output file in jobdir/shared/ unknown >> >> Any comments will be appreciated. >> >> best wishes >> zhangzhao >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >> >> >> > > From benc at hawaga.org.uk Thu Nov 13 00:19:17 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 13 Nov 2008 06:19:17 +0000 (GMT) Subject: [Swift-devel] discussion for swift output file path In-Reply-To: <491BC45C.4000502@uchicago.edu> References: <4913560A.7050000@uchicago.edu> <491BC45C.4000502@uchicago.edu> Message-ID: On Thu, 13 Nov 2008, Zhao Zhang wrote: > Are you working at CI tomorrow? I mean Thursday. So we could work together to > make this working. Yes, I am planning on going to the CI tomorrow. We can talk about stuff then. -- From zhaozhang at uchicago.edu Thu Nov 13 00:20:47 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 13 Nov 2008 00:20:47 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: References: <4913560A.7050000@uchicago.edu> <491BC45C.4000502@uchicago.edu> Message-ID: <491BC73F.7000208@uchicago.edu> cool. see you then. zhao Ben Clifford wrote: > On Thu, 13 Nov 2008, Zhao Zhang wrote: > > >> Are you working at CI tomorrow? I mean Thursday. So we could work together to >> make this working. >> > > Yes, I am planning on going to the CI tomorrow. We can talk about stuff > then. > > From zhaozhang at uchicago.edu Thu Nov 13 11:40:22 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 13 Nov 2008 11:40:22 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: References: <4913560A.7050000@uchicago.edu> <491BC45C.4000502@uchicago.edu> Message-ID: <491C6686.1070808@uchicago.edu> Ha, I make it working! cheers zhao Ben Clifford wrote: > On Thu, 13 Nov 2008, Zhao Zhang wrote: > > >> Are you working at CI tomorrow? I mean Thursday. So we could work together to >> make this working. >> > > Yes, I am planning on going to the CI tomorrow. We can talk about stuff > then. > > From benc at hawaga.org.uk Thu Nov 13 12:37:02 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 13 Nov 2008 18:37:02 +0000 (GMT) Subject: [Swift-devel] discussion for swift output file path In-Reply-To: <4913560A.7050000@uchicago.edu> References: <4913560A.7050000@uchicago.edu> Message-ID: On Thu, 6 Nov 2008, Zhao Zhang wrote: > I am working on integrate the Collective IO system and swift on BGP. Before > that, for the purpose of put swift into production work, It would be useful if the changes, if they're useful, end up in the production Swift SVN - keeping everyone's codebase fairly tightly converged seems a good thing to me. Points 1 and 3 below seem more difficult, but point 2: > 2. change wrapper.sh: copy files from local ramdisck to GPFS using dd > instead of cp, straghtforward. is easy to put in. Is there a measurable performance increase using dd rather than cp? And dd using default block size or different ones? > 1. change vdl-int.k: create hierarchical directory in shared/, > straightforward > 3. change somewhere in swift to make swift know where the data is, the path of > the output file in jobdir/shared/ unknown -- From zhaozhang at uchicago.edu Thu Nov 13 12:40:31 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 13 Nov 2008 12:40:31 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: References: <4913560A.7050000@uchicago.edu> Message-ID: <491C749F.5010500@uchicago.edu> Ben Clifford wrote: > On Thu, 6 Nov 2008, Zhao Zhang wrote: > > >> I am working on integrate the Collective IO system and swift on BGP. Before >> that, for the purpose of put swift into production work, >> > > It would be useful if the changes, if they're useful, end up in the > production Swift SVN - keeping everyone's codebase fairly tightly > converged seems a good thing to me. > > Points 1 and 3 below seem more difficult, but point 2: > > >> 2. change wrapper.sh: copy files from local ramdisck to GPFS using dd >> instead of cp, straghtforward. >> > > is easy to put in. Is there a measurable performance increase using dd > rather than cp? And dd using default block size or different ones? > yep, I made this working on surveyor. cp on CN uses a lined-buffer for output, thus a bad performance. the suggested block size of dd from Kamil is 128KB. zhao > >> 1. change vdl-int.k: create hierarchical directory in shared/, >> straightforward >> > > >> 3. change somewhere in swift to make swift know where the data is, the path of >> the output file in jobdir/shared/ unknown >> > > From benc at hawaga.org.uk Thu Nov 13 12:45:02 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 13 Nov 2008 18:45:02 +0000 (GMT) Subject: [Swift-devel] discussion for swift output file path In-Reply-To: <491C749F.5010500@uchicago.edu> References: <4913560A.7050000@uchicago.edu> <491C749F.5010500@uchicago.edu> Message-ID: On Thu, 13 Nov 2008, Zhao Zhang wrote: > yep, I made this working on surveyor. cp on CN uses a lined-buffer for output, > thus a bad performance. the suggested block size of dd from Kamil is 128KB. so you set that explicitly in the wrapper with a bs= option? -- From hategan at mcs.anl.gov Thu Nov 13 12:50:26 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 13 Nov 2008 12:50:26 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: <491C749F.5010500@uchicago.edu> References: <4913560A.7050000@uchicago.edu> <491C749F.5010500@uchicago.edu> Message-ID: <1226602226.30999.0.camel@localhost> On Thu, 2008-11-13 at 12:40 -0600, Zhao Zhang wrote: > > Ben Clifford wrote: > > On Thu, 6 Nov 2008, Zhao Zhang wrote: > > > > > >> I am working on integrate the Collective IO system and swift on BGP. Before > >> that, for the purpose of put swift into production work, > >> > > > > It would be useful if the changes, if they're useful, end up in the > > production Swift SVN - keeping everyone's codebase fairly tightly > > converged seems a good thing to me. > > > > Points 1 and 3 below seem more difficult, but point 2: > > > > > >> 2. change wrapper.sh: copy files from local ramdisck to GPFS using dd > >> instead of cp, straghtforward. > >> > > > > is easy to put in. Is there a measurable performance increase using dd > > rather than cp? And dd using default block size or different ones? > > > yep, I made this working on surveyor. cp on CN uses a lined-buffer for > output, thus a bad performance. Have you tried --sparse=never? From zhaozhang at uchicago.edu Thu Nov 13 13:13:25 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 13 Nov 2008 13:13:25 -0600 Subject: [Swift-devel] discussion for swift output file path In-Reply-To: References: <4913560A.7050000@uchicago.edu> <491C749F.5010500@uchicago.edu> Message-ID: <491C7C55.1050305@uchicago.edu> yes, dd if="$DIR/$O" of="$WFDIR/shared/$JOBDIR/$O" bs=128k zhao Ben Clifford wrote: > On Thu, 13 Nov 2008, Zhao Zhang wrote: > > >> yep, I made this working on surveyor. cp on CN uses a lined-buffer for output, >> thus a bad performance. the suggested block size of dd from Kamil is 128KB. >> > > so you set that explicitly in the wrapper with a bs= option? > > From zhaozhang at uchicago.edu Thu Nov 13 17:13:05 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 13 Nov 2008 17:13:05 -0600 Subject: [Swift-devel] exception report Message-ID: <491CB481.6070005@uchicago.edu> Hi, All I got an exception for the swift script on BGP, although it returned successful. The log file is at http://www.ci.uchicago.edu/~zzhang/dock3-20081113-1707-j38zbyp9.log The swift script is at http://www.ci.uchicago.edu/~zzhang/dock3.swift zhao zzhang at login6.surveyor:~/swift/test> ./run_swift.sh 0754 64 dock3.swift `pwd` waiting for at least 64 nodes to register before submitting workload... waiting to find at least 1 services in file /home/falkon/users/zzhang/0754/config/Client-service-URIs.config... all done, file has found at least 1 services found at least 64 registered, submitting workload... Swift svn swift-r2249 (Swift modified locally) cog-r2216 RunID: 20081113-1707-j38zbyp9 Progress: org.griphyn.vdl.karajan.VDL2FutureException at org.griphyn.vdl.mapping.MappingParam.checkHandle(MappingParam.java:74) at org.griphyn.vdl.mapping.MappingParam.getValue(MappingParam.java:46) at org.griphyn.vdl.mapping.MappingParam.getStringValue(MappingParam.java:82) at org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24) at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:67) at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:48) at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:38) at org.griphyn.vdl.karajan.lib.New.function(New.java:127) at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:65) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:283) at org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85) at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:45) at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.executeChildren(AbstractFunction.java:40) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java(Inlined Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled Code)) at org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled Code)) at org.globus.cog.karajan.workflow.events.EventWorker.run(EventWorker.java:69) From benc at hawaga.org.uk Thu Nov 13 17:14:50 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 13 Nov 2008 23:14:50 +0000 (GMT) Subject: [Swift-devel] exception report In-Reply-To: <491CB481.6070005@uchicago.edu> References: <491CB481.6070005@uchicago.edu> Message-ID: What modifications to swift did you have applied to this? On Thu, 13 Nov 2008, Zhao Zhang wrote: > Hi, All > > I got an exception for the swift script on BGP, although it returned > successful. > The log file is at > http://www.ci.uchicago.edu/~zzhang/dock3-20081113-1707-j38zbyp9.log > The swift script is at http://www.ci.uchicago.edu/~zzhang/dock3.swift > > zhao > > > zzhang at login6.surveyor:~/swift/test> ./run_swift.sh 0754 64 dock3.swift `pwd` > waiting for at least 64 nodes to register before submitting workload... > waiting to find at least 1 services in file > /home/falkon/users/zzhang/0754/config/Client-service-URIs.config... > all done, file has found at least 1 services > found at least 64 registered, submitting workload... > Swift svn swift-r2249 (Swift modified locally) cog-r2216 > > RunID: 20081113-1707-j38zbyp9 > Progress: > org.griphyn.vdl.karajan.VDL2FutureException > at > org.griphyn.vdl.mapping.MappingParam.checkHandle(MappingParam.java:74) > at org.griphyn.vdl.mapping.MappingParam.getValue(MappingParam.java:46) > at > org.griphyn.vdl.mapping.MappingParam.getStringValue(MappingParam.java:82) > at > org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24) > at > org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:67) > at > org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:48) > at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:38) > at org.griphyn.vdl.karajan.lib.New.function(New.java:127) > at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:65) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:283) > at > org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55) > at > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:45) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46) > at > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.executeChildren(AbstractFunction.java:40) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventWorker.run(EventWorker.java:69) > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From zhaozhang at uchicago.edu Thu Nov 13 17:20:48 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Thu, 13 Nov 2008 17:20:48 -0600 Subject: [Swift-devel] exception report In-Reply-To: References: <491CB481.6070005@uchicago.edu> Message-ID: <491CB650.4090708@uchicago.edu> Nothing new to the the data logic. All swift scripts which were working before are still working, we are just trying a new swfit script out. zhao Ben Clifford wrote: > What modifications to swift did you have applied to this? > > On Thu, 13 Nov 2008, Zhao Zhang wrote: > > >> Hi, All >> >> I got an exception for the swift script on BGP, although it returned >> successful. >> The log file is at >> http://www.ci.uchicago.edu/~zzhang/dock3-20081113-1707-j38zbyp9.log >> The swift script is at http://www.ci.uchicago.edu/~zzhang/dock3.swift >> >> zhao >> >> >> zzhang at login6.surveyor:~/swift/test> ./run_swift.sh 0754 64 dock3.swift `pwd` >> waiting for at least 64 nodes to register before submitting workload... >> waiting to find at least 1 services in file >> /home/falkon/users/zzhang/0754/config/Client-service-URIs.config... >> all done, file has found at least 1 services >> found at least 64 registered, submitting workload... >> Swift svn swift-r2249 (Swift modified locally) cog-r2216 >> >> RunID: 20081113-1707-j38zbyp9 >> Progress: >> org.griphyn.vdl.karajan.VDL2FutureException >> at >> org.griphyn.vdl.mapping.MappingParam.checkHandle(MappingParam.java:74) >> at org.griphyn.vdl.mapping.MappingParam.getValue(MappingParam.java:46) >> at >> org.griphyn.vdl.mapping.MappingParam.getStringValue(MappingParam.java:82) >> at >> org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24) >> at >> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:67) >> at >> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:48) >> at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:38) >> at org.griphyn.vdl.karajan.lib.New.function(New.java:127) >> at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:65) >> at >> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:283) >> at >> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45) >> at >> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:45) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31) >> at >> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.executeChildren(AbstractFunction.java:40) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventWorker.run(EventWorker.java:69) >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >> >> >> > > From benc at hawaga.org.uk Fri Nov 14 11:06:09 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 14 Nov 2008 17:06:09 +0000 (GMT) Subject: [Swift-devel] exception report In-Reply-To: <491CB650.4090708@uchicago.edu> References: <491CB481.6070005@uchicago.edu> <491CB650.4090708@uchicago.edu> Message-ID: On Thu, 13 Nov 2008, Zhao Zhang wrote: > Nothing new to the the data logic. All swift scripts which were working before > are still working, we are just trying a new swfit script out. Here is a useful debugging technique: Symptoms: I changed X and only X. Now A is broken. action: change X back to what it was before and see if A is broken. That's a very basic way of determining if the changes you made caused the problem. In your case, you claim that the only change is trying to run a new swiftscript. So if you run an old swiftscript that worked a day or so ago, what happens? Do you get the error or not? When reporting problems, its useful to try things like this first, and include the information about where X really does afffect A in your problem report email. It makes it a lot easier for other people to think about narrowing down the problem. -- From benc at hawaga.org.uk Fri Nov 14 11:09:03 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 14 Nov 2008 17:09:03 +0000 (GMT) Subject: [Swift-devel] exception report In-Reply-To: <491CB481.6070005@uchicago.edu> References: <491CB481.6070005@uchicago.edu> Message-ID: Hmm. Can you send the source code for your SwiftScript? I think you might be running into the 'data dependency vs sequential execution' problem that I've been working to fix recently. But I'd lke to see your source code. On Thu, 13 Nov 2008, Zhao Zhang wrote: > Hi, All > > I got an exception for the swift script on BGP, although it returned > successful. > The log file is at > http://www.ci.uchicago.edu/~zzhang/dock3-20081113-1707-j38zbyp9.log > The swift script is at http://www.ci.uchicago.edu/~zzhang/dock3.swift > > zhao > > > zzhang at login6.surveyor:~/swift/test> ./run_swift.sh 0754 64 dock3.swift `pwd` > waiting for at least 64 nodes to register before submitting workload... > waiting to find at least 1 services in file > /home/falkon/users/zzhang/0754/config/Client-service-URIs.config... > all done, file has found at least 1 services > found at least 64 registered, submitting workload... > Swift svn swift-r2249 (Swift modified locally) cog-r2216 > > RunID: 20081113-1707-j38zbyp9 > Progress: > org.griphyn.vdl.karajan.VDL2FutureException > at > org.griphyn.vdl.mapping.MappingParam.checkHandle(MappingParam.java:74) > at org.griphyn.vdl.mapping.MappingParam.getValue(MappingParam.java:46) > at > org.griphyn.vdl.mapping.MappingParam.getStringValue(MappingParam.java:82) > at > org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24) > at > org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:67) > at > org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:48) > at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:38) > at org.griphyn.vdl.karajan.lib.New.function(New.java:127) > at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:65) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:283) > at > org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55) > at > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:45) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46) > at > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.executeChildren(AbstractFunction.java:40) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java(Inlined > Compiled Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled > Code)) > at > org.globus.cog.karajan.workflow.events.EventWorker.run(EventWorker.java:69) > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From zhaozhang at uchicago.edu Fri Nov 14 13:07:26 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Fri, 14 Nov 2008 13:07:26 -0600 Subject: [Swift-devel] exception report In-Reply-To: References: <491CB481.6070005@uchicago.edu> Message-ID: <491DCC6E.7090008@uchicago.edu> Hi, Sorry for the late response, the source code is here: http://www.ci.uchicago.edu/~zzhang/dock3.swift And yes, my other swift scripts work well. zhao Ben Clifford wrote: > Hmm. Can you send the source code for your SwiftScript? > > I think you might be running into the 'data dependency vs sequential > execution' problem that I've been working to fix recently. But I'd lke to > see your source code. > > On Thu, 13 Nov 2008, Zhao Zhang wrote: > > >> Hi, All >> >> I got an exception for the swift script on BGP, although it returned >> successful. >> The log file is at >> http://www.ci.uchicago.edu/~zzhang/dock3-20081113-1707-j38zbyp9.log >> The swift script is at http://www.ci.uchicago.edu/~zzhang/dock3.swift >> >> zhao >> >> >> zzhang at login6.surveyor:~/swift/test> ./run_swift.sh 0754 64 dock3.swift `pwd` >> waiting for at least 64 nodes to register before submitting workload... >> waiting to find at least 1 services in file >> /home/falkon/users/zzhang/0754/config/Client-service-URIs.config... >> all done, file has found at least 1 services >> found at least 64 registered, submitting workload... >> Swift svn swift-r2249 (Swift modified locally) cog-r2216 >> >> RunID: 20081113-1707-j38zbyp9 >> Progress: >> org.griphyn.vdl.karajan.VDL2FutureException >> at >> org.griphyn.vdl.mapping.MappingParam.checkHandle(MappingParam.java:74) >> at org.griphyn.vdl.mapping.MappingParam.getValue(MappingParam.java:46) >> at >> org.griphyn.vdl.mapping.MappingParam.getStringValue(MappingParam.java:82) >> at >> org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24) >> at >> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:67) >> at >> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:48) >> at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:38) >> at org.griphyn.vdl.karajan.lib.New.function(New.java:127) >> at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:65) >> at >> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:283) >> at >> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45) >> at >> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:45) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31) >> at >> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.executeChildren(AbstractFunction.java:40) >> at >> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java(Inlined >> Compiled Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java(Compiled >> Code)) >> at >> org.globus.cog.karajan.workflow.events.EventWorker.run(EventWorker.java:69) >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >> >> >> > > From benc at hawaga.org.uk Fri Nov 14 13:12:19 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 14 Nov 2008 19:12:19 +0000 (GMT) Subject: [Swift-devel] exception report In-Reply-To: <491DCC6E.7090008@uchicago.edu> References: <491CB481.6070005@uchicago.edu> <491DCC6E.7090008@uchicago.edu> Message-ID: On Fri, 14 Nov 2008, Zhao Zhang wrote: > Hi, Sorry for the late response, the source code is here: > http://www.ci.uchicago.edu/~zzhang/dock3.swift I think this will not work: string mname=p.mname; Ligand mfile ; You should be able to remove those two lines and say: Ligand mfile ; -- From zhaozhang at uchicago.edu Fri Nov 14 13:14:32 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Fri, 14 Nov 2008 13:14:32 -0600 Subject: [Swift-devel] exception report In-Reply-To: References: <491CB481.6070005@uchicago.edu> <491DCC6E.7090008@uchicago.edu> Message-ID: <491DCE18.1030603@uchicago.edu> cool, thanks, Ben, I will try this out. zhao Ben Clifford wrote: > On Fri, 14 Nov 2008, Zhao Zhang wrote: > > >> Hi, Sorry for the late response, the source code is here: >> http://www.ci.uchicago.edu/~zzhang/dock3.swift >> > > I think this will not work: > > string mname=p.mname; > Ligand mfile ; > > You should be able to remove those two lines and say: > > Ligand mfile ; > > From wilde at mcs.anl.gov Fri Nov 14 13:27:10 2008 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 14 Nov 2008 13:27:10 -0600 Subject: [Swift-devel] exception report In-Reply-To: References: <491CB481.6070005@uchicago.edu> <491DCC6E.7090008@uchicago.edu> Message-ID: <491DD10E.2080201@mcs.anl.gov> Can you explain why, Ben? On 11/14/08 1:12 PM, Ben Clifford wrote: > On Fri, 14 Nov 2008, Zhao Zhang wrote: > >> Hi, Sorry for the late response, the source code is here: >> http://www.ci.uchicago.edu/~zzhang/dock3.swift > > I think this will not work: > > string mname=p.mname; > Ligand mfile ; > > You should be able to remove those two lines and say: > > Ligand mfile ; > From benc at hawaga.org.uk Fri Nov 14 13:35:39 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 14 Nov 2008 19:35:39 +0000 (GMT) Subject: [Swift-devel] exception report In-Reply-To: <491DD10E.2080201@mcs.anl.gov> References: <491CB481.6070005@uchicago.edu> <491DCC6E.7090008@uchicago.edu> <491DD10E.2080201@mcs.anl.gov> Message-ID: On Fri, 14 Nov 2008, Michael Wilde wrote: > Can you explain why, Ben? It looks like the same 'dataflow ordering that isn't fully implemented' problem that often comes up - if so, it should be solved by the stuff I'm working on at the moment. Mappers can't take anything as a parameter that comes from "something complicated" (where thats a deliberatedly vague term hiding many details of the insides of the present runtime). -- From iraicu at cs.uchicago.edu Fri Nov 14 16:34:03 2008 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Fri, 14 Nov 2008 16:34:03 -0600 Subject: [Swift-devel] Falkon related activities at SC08 Message-ID: <491DFCDB.8040706@cs.uchicago.edu> Hi all, I setup a wiki (http://dev.globus.org/wiki/SC08_Activities) outlining the various talks or events that are taking place next week at SC08 that might be of interest. Hope to see some of you there! Cheers, Ioan -- =================================================== Ioan Raicu Ph.D. Candidate =================================================== Distributed Systems Laboratory Computer Science Department University of Chicago 1100 E. 58th Street, Ryerson Hall Chicago, IL 60637 =================================================== Email: iraicu at cs.uchicago.edu Web: http://www.cs.uchicago.edu/~iraicu http://dev.globus.org/wiki/Incubator/Falkon http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page =================================================== =================================================== From bugzilla-daemon at mcs.anl.gov Sat Nov 15 19:55:21 2008 From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov) Date: Sat, 15 Nov 2008 19:55:21 -0600 (CST) Subject: [Swift-devel] [Bug 158] New: Tidy up backslash escape processing Message-ID: http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=158 Summary: Tidy up backslash escape processing Product: Swift Version: unspecified Platform: Macintosh OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: benc at hawaga.org.uk ReportedBy: wilde at mcs.anl.gov CC: benc at hawaga.org.uk On Sat, 15 Nov 2008, Michael Wilde wrote: > > It seems from experimentation that backslash quoting changed at some point (or > > the user guide and tutorial are wrong): the \1 needed in the transform arg of > > the regexp_mapper needs to be \\1 - at least it seems this way from > > experimentation with trace(). Yes, that changed between v0.4 and v0.5. This is an intended behaviour, to facilitate putting in \" characters. > > Its also odd trace() takes \t and \n but echoes them just like that, with the > > backslashes printing. I don't think has had any strong intention about that should behave. On looking at the parser, \ followed by one of n r t b f " \ will be passed through, with apparently only the quotes having some special behaviour and the others passing through unmodified. Other \ sequences, such as \g, will cause a parser error. This could be tidied up sometime. File a bug for it. -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You reported the bug, or are watching the reporter. From benc at hawaga.org.uk Sun Nov 16 01:37:12 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Sun, 16 Nov 2008 07:37:12 +0000 (GMT) Subject: [Swift-devel] karajan futures and getfieldvalue/setfieldvalue Message-ID: While playing round with how setfieldvalue works, I came across the following; I don't understand enough about how karajan deals / is meant to deal with futures to see what is going on. The attached attach, error.patch, is intended to amend the behaviour of Swift HEAD so that getFieldValue is not used to get an unwrapped (not-in-DSHandle) value to pass into SetFieldValue. The attached patch test.patch contains a test, test.swift, which appears to work repeatedly on swift/cog HEAD but fails often on my laptop with error.patch applied (although sometimes works). Changing the number of assignment steps in test.swift changes the likelihood that the execution will fail - more steps = higher probability of failure (which makes sense as there is a race to get the circumstances that trigger this). It looks like if SetFieldValue throws a FutureNotYetAvailable exception, then this does not have the same behaviour of deferring execution and rerunning later, as when GetFieldValue throws a FutureNotAvailable exception which seems to work ok. There is an execution log for a failed log at http://www.ci.uchicago.edu/~benc/test-20081116-0133-vr84izb8.log (btw When running this test on two different versions, don't forget to force a recompile: touch test.swift && swift test.swift) -- -------------- next part -------------- A non-text attachment was scrubbed... Name: error2.patch Type: text/x-diff Size: 2366 bytes Desc: URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.patch Type: text/x-diff Size: 307 bytes Desc: URL: From hategan at mcs.anl.gov Sun Nov 16 11:13:47 2008 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 16 Nov 2008 11:13:47 -0600 Subject: [Swift-devel] Re: karajan futures and getfieldvalue/setfieldvalue In-Reply-To: References: Message-ID: <1226855627.11651.5.camel@localhost> Maybe you're synchronizing on the wrong thing there? I'd try value instead of value.getRoot(). That's a suspicion. I'll have to check though. Do you have the complete set of patches? On Sun, 2008-11-16 at 07:37 +0000, Ben Clifford wrote: > While playing round with how setfieldvalue works, I came across the > following; I don't understand enough about how karajan deals / is meant to > deal with futures to see what is going on. > > The attached attach, error.patch, is intended to amend the behaviour of > Swift HEAD so that getFieldValue is not used to get an unwrapped > (not-in-DSHandle) value to pass into SetFieldValue. > > The attached patch test.patch contains a test, test.swift, which appears > to work repeatedly on swift/cog HEAD but fails often on my laptop with > error.patch applied (although sometimes works). > > Changing the number of assignment steps in test.swift changes the > likelihood that the execution will fail - more steps = higher probability > of failure (which makes sense as there is a race to get the circumstances > that trigger this). > > It looks like if SetFieldValue throws a FutureNotYetAvailable exception, > then this does not have the same behaviour of deferring execution and > rerunning later, as when GetFieldValue throws a FutureNotAvailable > exception which seems to work ok. > > There is an execution log for a failed log at > http://www.ci.uchicago.edu/~benc/test-20081116-0133-vr84izb8.log > > (btw When running this test on two different versions, don't forget to > force a recompile: touch test.swift && swift test.swift) > From benc at hawaga.org.uk Sun Nov 16 15:48:53 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Sun, 16 Nov 2008 21:48:53 +0000 (GMT) Subject: [Swift-devel] Re: karajan futures and getfieldvalue/setfieldvalue In-Reply-To: <1226855627.11651.5.camel@localhost> References: <1226855627.11651.5.camel@localhost> Message-ID: On Sun, 16 Nov 2008, Mihael Hategan wrote: > That's a suspicion. I'll have to check though. Do you have the complete > set of patches? That is the only patch - special pristine. -- From benc at hawaga.org.uk Mon Nov 17 09:06:22 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 17 Nov 2008 15:06:22 +0000 (GMT) Subject: [Swift-devel] Re: karajan futures and getfieldvalue/setfieldvalue In-Reply-To: <1226855627.11651.5.camel@localhost> References: <1226855627.11651.5.camel@localhost> Message-ID: Elaborating on my rather terse reply yesterday: On Sun, 16 Nov 2008, Mihael Hategan wrote: > Maybe you're synchronizing on the wrong thing there? I'd try value > instead of value.getRoot(). I've been fiddling elsewhere with locking in order to fix up some deadlocks I ended up getting - synchronizing on the root of structures instead of on pieces of that is something I've been playing with. > That's a suspicion. I'll have to check though. Do you have the complete > set of patches? The patch I sent is sufficient to apply to Swift HEAD to demonstrate the bug without needing anything else - I have a large patchset with a lot of other stuff going on, but this patch is enough for me to get this happening without other patches applied. -- From benc at hawaga.org.uk Mon Nov 17 09:10:47 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 17 Nov 2008 15:10:47 +0000 (GMT) Subject: [Swift-devel] Re: karajan futures and getfieldvalue/setfieldvalue In-Reply-To: References: <1226855627.11651.5.camel@localhost> Message-ID: Changing the lock to lock on 'value' rather than 'value.getRoot()' shows the same (or similar) problem. -- From zhaozhang at uchicago.edu Mon Nov 17 16:53:42 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Mon, 17 Nov 2008 16:53:42 -0600 Subject: [Swift-devel] log plotting for 3-stage dock6 run with swift on surveyor Message-ID: <4921F5F6.1030902@uchicago.edu> Hi, All Here is the link http://www.ci.uchicago.edu/~zzhang/report-demo-dock7-20081117-1605-13f4ut97/ Could we tell something from this plotting that we could improve the swift performance? Thanks best wishes zhangzhao From zhaozhang at uchicago.edu Mon Nov 17 17:11:37 2008 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Mon, 17 Nov 2008 17:11:37 -0600 Subject: [Swift-devel] Re: log plotting for 3-stage dock6 run with swift on surveyor In-Reply-To: <4921F5F6.1030902@uchicago.edu> References: <4921F5F6.1030902@uchicago.edu> Message-ID: <4921FA29.8070609@uchicago.edu> Hi, Again Here is more additional information: Basically we run 2048 dock6 jobs among 2048 cores, then another 2048 jobs to analyze the previous outputs. Finally we select some of those 2048 dock6 result files. The swift script is at http://www.ci.uchicago.edu/~zzhang/demo-dock7.swift each line of paramlist file is like this: 0 /home/ligandatlas/databases/KEGG_and_Drugs/C00001.mol2 000/000/run01_out.0000000.tar.gz CAPD Parameters in swift.properties are : throttle.transfers=off throttle.file.operations=off jobThrottle=8 initialScore=1000 Is there anything I could to improve the performance? Thanks zhao Zhao Zhang wrote: > Hi, All > > Here is the link > http://www.ci.uchicago.edu/~zzhang/report-demo-dock7-20081117-1605-13f4ut97/ > > > Could we tell something from this plotting that we could improve the > swift performance? Thanks > > best wishes > zhangzhao > From aespinosa at cs.uchicago.edu Fri Nov 21 17:35:41 2008 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Fri, 21 Nov 2008 17:35:41 -0600 Subject: [Swift-devel] Re: BLAST on Swift In-Reply-To: <50b07b4b0811151310v6bb08d4dq331a47d059e92c73@mail.gmail.com> References: <50b07b4b0811151310v6bb08d4dq331a47d059e92c73@mail.gmail.com> Message-ID: <50b07b4b0811211535g19ad63e3j8c1fa95f14e9a777@mail.gmail.com> Hi, Here is a modification of the blast swiftscript: type database; type query; type output; type error; (output out, error err) blastall(query i, database pref, database db[]) { app { blastall "-p" "blastp" "-F" "F" "-d" @filename(pref) "-i" @filename(i) "-v" "300" "-b" "300" "-m8" "-o" @filename(out) stderr=@filename(err); } } database pirpref <"/disks/ci-gpfs/swift/blast/pir/UNIPROT_for_blast_14.0.seq">; database pir[] ; output out <"test.out">; query i <"test.in">; error err <"test.err">; (out,err) = blastall(i, pirpref, pir); The blastall binary tries to implement its own data management so we specify in the commandline only the *prefix* name of the database. I made a zero-size dummy file named $PREFIX in the directory of the input database files. Then i pass the entire array to the function to satisfy data dependency. I noticed that "ls | sort" on the directory will return $PREFIX first in the list of files. Swift on the other hand returns it as a last element. I'd like to try this script out on Ranger for a full blast run. Anyone have preconfigured sites.xml for this system? Thanks, -Allan On Sat, Nov 15, 2008 at 3:10 PM, Allan Espinosa wrote: > Hi all, > > I was able to run Blast locally on communicado today. The swift > script basically needs the query an database file passed to the atomic > function. It then returns the resultant sequence and stderr logs. > > Below is the swift script: > ---start--- > type database; > type query; > type output; > type error; > > (output out, error err) blastall(query i, database db) { > app { > blastall "-p" "blastp" "-F" "F" "-d" @filename(db) "-i" > @filename(i) "-v" "300" "-b" "300" "-m8" "-o" @filename(out) > stderr=@filename(err); > } > } > > database pir ; > output out <"test.out">; > query i <"test.in">; > error err <"test.err">; > (out,err) = blastall(i, pir); > ---end--- > > I looked at the dock6 documentation for OSG. It looks that it > recommends to transfer the datafiles to OSG sites manually via > globus-url-copy. By my understanding of how swift works, it should be > able to transfer my local files to the selected sites. I have yet to > try this and will look more on examples in the data management side of > Swift. > > Do you know other users who went in this approach? The documentation > has only a few examples in managing data. I'll check the swift Wiki > later and see what material we have and also post this email/ notes. From benc at hawaga.org.uk Fri Nov 21 23:01:13 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 22 Nov 2008 05:01:13 +0000 (GMT) Subject: [Swift-devel] Re: BLAST on Swift In-Reply-To: <50b07b4b0811211535g19ad63e3j8c1fa95f14e9a777@mail.gmail.com> References: <50b07b4b0811151310v6bb08d4dq331a47d059e92c73@mail.gmail.com> <50b07b4b0811211535g19ad63e3j8c1fa95f14e9a777@mail.gmail.com> Message-ID: > I'd like to try this script out on Ranger for a full blast run. > Anyone have preconfigured sites.xml for this system? skenny gave me this the other day. It might work for you: 2 1 16 /work/00640/tg458015 -- From benc at hawaga.org.uk Sun Nov 23 01:15:56 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Sun, 23 Nov 2008 07:15:56 +0000 (GMT) Subject: [Swift-devel] LONI Pipeline -> Swift conversion Message-ID: I had a play today with a converter from LONI Pipeline saved workflows to Swift. This is to see how well Swift can be used as an execution layer for LONI Pipeline workflows. So today I can convert a simple 'one input file - one execution - one output file' workflow and run it through Swift. -- From benc at hawaga.org.uk Sun Nov 23 19:22:43 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 24 Nov 2008 01:22:43 +0000 (GMT) Subject: [Swift-devel] Re: karajan futures and getfieldvalue/setfieldvalue In-Reply-To: References: Message-ID: On Sun, 16 Nov 2008, Ben Clifford wrote: > It looks like if SetFieldValue throws a FutureNotYetAvailable exception, > then this does not have the same behaviour of deferring execution and > rerunning later, as when GetFieldValue throws a FutureNotAvailable > exception which seems to work ok. I played with this today and made it go away. SetFieldValue was actually incorrectly catching the FutureNotYetAvailable exception (as part of catching all Exceptions) and then throwing an ExecutionException with no message set. That ExecutionException was displayed in a way that is confusingly like the exception it wraps. A simple fix was to make this catch FutureNotYetAvailable and rethrow it. Probably the catch of all Exceptions should be tightened up. I'll commit code for this later today. -- From benc at hawaga.org.uk Mon Nov 24 16:59:31 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 24 Nov 2008 22:59:31 +0000 (GMT) Subject: [Swift-devel] date of Swift 0.8 release Message-ID: A release every two months puts 0.8 sometime in mid-January. So that's when I plan to make it. -- From foster at cs.uchicago.edu Tue Nov 25 12:50:02 2008 From: foster at cs.uchicago.edu (Ian Foster) Date: Tue, 25 Nov 2008 12:50:02 -0600 Subject: [Swift-devel] Fwd: Call For Papers: Intl. Workshop on Parallel Programming Models and Systems Software for HEC (P2S2) References: <200811251750.mAPHovT3002953@pakkled.iit.edu> Message-ID: <48B84CC9-379C-4ABF-ACB7-282F3B519CA7@cs.uchicago.edu> A potential workshop for a Swift paper. Begin forwarded message: > From: p2s2-chairs at mcs.anl.gov > Date: November 25, 2008 11:50:57 AM CST > To: undisclosed-recipients: ; > Subject: Call For Papers: Intl. Workshop on Parallel Programming > Models and Systems Software for HEC (P2S2) > Reply-To: p2s2-chairs at mcs.anl.gov > > CALL FOR PAPERS > =============== > > Second International Workshop on Parallel Programming Models > and Systems Software for High-end Computing (P2S2) > Sept. 22nd, 2009 > > To be held in conjunction with ICPP-09: The 38th International > Conference on Parallel Processing, Sept. 22-25, 2009, Vienna, Austria > > Website: http://www.mcs.anl.gov/events/workshops/p2s2 > > SCOPE > ----- > The goal of this workshop is to bring together researchers and > practitioners in parallel programming models and systems software for > high-end computing systems. Please join us in a discussion of new > ideas, > experiences, and the latest trends in these areas at the workshop. > > > TOPICS OF INTEREST > ------------------ > The focus areas for this workshop include, but are not limited to: > > * Systems software for high-end scientific and enterprise > computing architectures > o Communication sub-subsystems for high-end computing > o High-performance file and storage systems > o Fault-tolerance techniques and implementations > o Efficient and high-performance virtualization and other > management mechanisms for high-end computing > > * Programming models and their high-performance implementations > o MPI, Sockets, OpenMP, Global Arrays, X10, UPC, Chapel, > Fortress and others > o Hybrid Programming Models > > * Tools for Management, Maintenance, Coordination and > Synchronization > o Software for Enterprise Data-centers using Modern > Architectures > o Job scheduling libraries > o Management libraries for large-scale system > o Toolkits for process and task coordination on modern > platforms > > * Performance evaluation, analysis and modeling of emerging > computing platforms > > > PROCEEDINGS > ----------- > Proceedings of this workshop will be published by the IEEE Computer > Society (together with the ICPP conference proceedings) in CD format > only and will be available at the conference. > > > SUBMISSION INSTRUCTIONS > ----------------------- > Submissions should be in PDF format in U.S. Letter size paper. They > should not exceed 8 pages (all inclusive). Submissions will be judged > based on relevance, significance, originality, correctness and > clarity. > Please visit workshop website at: http://www.mcs.anl.gov/events/workshops/p2s2/ > for the submission link. > > > JOURNAL SPECIAL ISSUE > --------------------- > The best papers selected for the workshop will be published in a > special issue of the International Journal of High Performance > Computing Applications (IJHPCA) on Parallel Programming Models and > Systems Software for High-End Computing. > > > IMPORTANT DATES > --------------- > Paper Submission: Feb. 27th, 2009 > Author Notification: May 1st, 2009 > Camera Ready: June 5th, 2009 > > > PROGRAM CHAIRS > -------------- > * Pavan Balaji (Argonne National Laboratory) > * Abhinav Vishnu (Pacific Northwest National Laboratory) > > > PUBLICITY CHAIR > --------------- > * Yong Chen, Illinois Institute of Technology > > > STEERING COMMITTEE > ------------------ > * William D. Gropp (University of Illinois Urbana-Champaign) > * Dhabaleswar K. Panda (Ohio State University) > * Vijay Saraswat (IBM Research) > > > PROGRAM COMMITTEE > ----------------- > * Taisuke Boku, University of Tsukuba, Japan > * Ron Brightwell, Sandia National Laboratory > * Narayan Desai, Argonne National Laboratory > * Richard Graham, Oak Ridge National Laboratory > * Zhiyi Huang, University of Otago, New Zealand > * Hyun-Wook Jin, Konkuk University, Korea > * Matthew Koop, Ohio State University > * Sriram Krishnamoorthy, Pacific Northwest National Laboratory > * Zhiling Lan, Illinois Institute of Technology > * Doug Lea, State University of New York at Oswego > * Jiuxing Liu, IBM Research > * Guillaume Mercier, INRIA, France > * Jarek Nieplocha, Pacific Northwest National Laboratory > * Scott Pakin, Los Alamos National Laboratory > * Fabrizio Petrini, IBM Research > * Arun Raghunath, Intel > * Vivek Sarkar, Rice University > * Bronis de Supinksi, Lawrence Livermore National Laboratory > * Sayantan Sur, IBM Research > * Rajeev Thakur, Argonne National Laboratory > * Jesper Traff, NEC, Europe > * Weikuan Yu, Oak Ridge National Laboratory > > > If you have any questions, please contact us at p2s2- > chairs at mcs.anl.gov > > = > = > ====================================================================== > If you do not want to receive any more announcements regarding the > P2S2 workshop, please unsubscribe here: > https://lists.mcs.anl.gov/mailman/listinfo/p2s2-announce > = > = > ====================================================================== > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benc at hawaga.org.uk Sat Nov 29 16:45:32 2008 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 29 Nov 2008 22:45:32 +0000 (GMT) Subject: [Swift-devel] Functional building blocks as concurrency patterns Message-ID: This just appeared on lambda-the-ultimate: http://lambda-the-ultimate.org/node/3108 > While teaching INGI1131, my concurrent programming course, I have become > even more impressed by a concurrent paradigm, namely functional > programming extended with threads and ports, which I call multi-agent > dataflow programming. Some ofthe concepts overlap or are closely related to SwiftScript and so some of the discussion on that post is interesting to skim. Some differences are that: * we don't have infinite/unbounded streams of data because of the way that our mappers work at the moment (though it conceptually fits in with SwiftScript); and * we don't have ports (or some other way of introducing non-determininism in the language itself) - we do have something similar at the execution layer, though, in the form of job replication, where we allow two job submissions to race to start execution and only keep the one that starts executing (some day it might be desirable to extend this to the one that finishes first or some other more complex expression, but I don't see need at the moment) --