From tiberius at ci.uchicago.edu Tue Apr 3 10:25:01 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Tue, 3 Apr 2007 10:25:01 -0500 Subject: [Swift-devel] Swift Question/Request Message-ID: Sometimes i find myself in the position of wanting to do one step of the swift script only after the previous phase has finished. The case is the following (not really swift script): seq{ file stageOneOutputs[] = processStageOne(); file combine = combine(stageOneOutputs); } In my case the problem seems to be the fact that swift cannot dream up how many stageOneOutput files will exist (unless we extend the language -probably the input mapper which I use in this case- to support this feature), so it could decide to start the combine stage prematurely. Supposedly one solution is to create a dummy output file at the end of stage one, and pass it tot stage two. Any suggestions ? I'm thinking that having an example in the user guide would help others as well. Maybe we should have a "Patterns in Swift" document. Thanks Tibi -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From yongzh at cs.uchicago.edu Tue Apr 3 10:42:58 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Tue, 3 Apr 2007 10:42:58 -0500 (CDT) Subject: [Swift-devel] Swift Question/Request In-Reply-To: References: Message-ID: Can you be more specific about why it can't figure out the number of outputs correctly? The sequential order should already be defined in your example as there is explicit data dependency. Yong. On Tue, 3 Apr 2007, Tiberiu Stef-Praun wrote: > Sometimes i find myself in the position of wanting to do one step of > the swift script only after the previous phase has finished. The case > is the following (not really swift script): > > seq{ > file stageOneOutputs[] = processStageOne(); > file combine = combine(stageOneOutputs); > } > > In my case the problem seems to be the fact that swift cannot dream up > how many stageOneOutput files will exist (unless we extend the > language -probably the input mapper which I use in this case- to > support this feature), so it could decide to start the combine stage > prematurely. > Supposedly one solution is to create a dummy output file at the end of > stage one, and pass it tot stage two. > > Any suggestions ? > > I'm thinking that having an example in the user guide would help others as well. > Maybe we should have a "Patterns in Swift" document. > > Thanks > Tibi > > -- > Tiberiu (Tibi) Stef-Praun, PhD > Research Staff, Computation Institute > 5640 S. Ellis Ave, #405 > University of Chicago > http://www-unix.mcs.anl.gov/~tiberius/ > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From tiberius at ci.uchicago.edu Tue Apr 3 10:52:08 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Tue, 3 Apr 2007 10:52:08 -0500 Subject: [Swift-devel] Swift Question/Request In-Reply-To: References: Message-ID: I'm not sure that the workflow below will wok as intended. The uncertainty I have is after the submission to execution of the moralhazard_problem procedure. I have not tested it completely (I only ran it up to the moralhazard_problem invocation), I'm still working on the mergeSolutions script Tibi ===================================== type file{} //define the linear solver (file solutions) moralhazard_solver (file scriptfile, int batch_size, int batch_no, file init_iteration, file init_param, file init_limits, file init_surplus){ app{ moralhazard @filename(scriptfile) batch_size batch_no; } } //(file solutions[]) moralhazard_problem (file inputfiles[]){ moralhazard_problem (file inputfiles[]){ file script<"scripts/moralhazard-param.m">; file iteration<"iteration.mat">; file param<"param.mat">; file limits<"limits.mat">; file surplus<"Surplus.mat">; // the limit of 36 needs to be adjusted // the batch_size can change, together with the limit // batch_size x batch_limit = gridsize(26x26)/2 int batch_limit=36; int batch_size=10; int batch_range=[0:35]; foreach i in batch_range { int position=i*batch_size; file solution; solution=moralhazard_solver(script,batch_size,position,iteration,param,limits,surplus); } } file problems[] ; file results[]; moralhazard_problem (problems); file solutions[] ; file solution <"mergedSolution.txt">; file solution combineSolutions(file solverSolutions[]){ app{ mergeSolutions @filenames(solverSolutions) stdout=@filename(solution); } } combineSolutions(solutions); On 4/3/07, Yong Zhao wrote: > Can you be more specific about why it can't figure out the number of > outputs correctly? The sequential order should already be defined in your > example as there is explicit data dependency. > > Yong. > > On Tue, 3 Apr 2007, Tiberiu Stef-Praun wrote: > > > Sometimes i find myself in the position of wanting to do one step of > > the swift script only after the previous phase has finished. The case > > is the following (not really swift script): > > > > seq{ > > file stageOneOutputs[] = processStageOne(); > > file combine = combine(stageOneOutputs); > > } > > > > In my case the problem seems to be the fact that swift cannot dream up > > how many stageOneOutput files will exist (unless we extend the > > language -probably the input mapper which I use in this case- to > > support this feature), so it could decide to start the combine stage > > prematurely. > > Supposedly one solution is to create a dummy output file at the end of > > stage one, and pass it tot stage two. > > > > Any suggestions ? > > > > I'm thinking that having an example in the user guide would help others as well. > > Maybe we should have a "Patterns in Swift" document. > > > > Thanks > > Tibi > > > > -- > > Tiberiu (Tibi) Stef-Praun, PhD > > Research Staff, Computation Institute > > 5640 S. Ellis Ave, #405 > > University of Chicago > > http://www-unix.mcs.anl.gov/~tiberius/ > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From hategan at mcs.anl.gov Tue Apr 3 10:58:49 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 03 Apr 2007 10:58:49 -0500 Subject: [Swift-devel] Swift Question/Request In-Reply-To: References: Message-ID: <1175615929.20544.5.camel@blabla.mcs.anl.gov> How exactly does seq make Swift know how many outputs processStageOne() has? On Tue, 2007-04-03 at 10:25 -0500, Tiberiu Stef-Praun wrote: > Sometimes i find myself in the position of wanting to do one step of > the swift script only after the previous phase has finished. The case > is the following (not really swift script): > > seq{ > file stageOneOutputs[] = processStageOne(); > file combine = combine(stageOneOutputs); > } > > In my case the problem seems to be the fact that swift cannot dream up > how many stageOneOutput files will exist (unless we extend the > language -probably the input mapper which I use in this case- to > support this feature), so it could decide to start the combine stage > prematurely. I seriously doubt that. It may not start it at all, but prematurely, no. > Supposedly one solution is to create a dummy output file at the end of > stage one, and pass it tot stage two. > > Any suggestions ? This kind of problem can only be addressed in two ways: 1. You know how many files you have when you write your program (and use a fixed array mapper). 2. Swift can figure out how many files are out there. This is something that, by virtue of the number of times we've seen this cropping and the fact that there is no hack that can solve it, should be on our list of things (if not already there). > > I'm thinking that having an example in the user guide would help others as well. > Maybe we should have a "Patterns in Swift" document. > > Thanks > Tibi > From tiberius at ci.uchicago.edu Tue Apr 3 11:08:58 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Tue, 3 Apr 2007 11:08:58 -0500 Subject: [Swift-devel] Swift Question/Request In-Reply-To: <1175615929.20544.5.camel@blabla.mcs.anl.gov> References: <1175615929.20544.5.camel@blabla.mcs.anl.gov> Message-ID: On 4/3/07, Mihael Hategan wrote: > How exactly does seq make Swift know how many outputs processStageOne() > has? > > On Tue, 2007-04-03 at 10:25 -0500, Tiberiu Stef-Praun wrote: > > Sometimes i find myself in the position of wanting to do one step of > > the swift script only after the previous phase has finished. The case > > is the following (not really swift script): > > > > seq{ > > file stageOneOutputs[] = processStageOne(); > > file combine = combine(stageOneOutputs); > > } > > > > In my case the problem seems to be the fact that swift cannot dream up > > how many stageOneOutput files will exist (unless we extend the > > language -probably the input mapper which I use in this case- to > > support this feature), so it could decide to start the combine stage > > prematurely. > > I seriously doubt that. It may not start it at all, but prematurely, no. > > > Supposedly one solution is to create a dummy output file at the end of > > stage one, and pass it tot stage two. > > > > Any suggestions ? > > This kind of problem can only be addressed in two ways: > 1. You know how many files you have when you write your program (and use > a fixed array mapper). What I had in mind was something along these lines: I know how many output files to expect, but It might be ugly specifying their names, so I was thinking of adding a regexp limit to the filesys mapper: In my case I know I am expecting 36 output files, so don't start the second stage untill the filesys mapper can produce those many files. But for now I will probably go with the fixed array mapper. > 2. Swift can figure out how many files are out there. This is something > that, by virtue of the number of times we've seen this cropping and the > fact that there is no hack that can solve it, should be on our list of > things (if not already there). > > > > > I'm thinking that having an example in the user guide would help others as well. > > Maybe we should have a "Patterns in Swift" document. > > > > Thanks > > Tibi > > > > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From hategan at mcs.anl.gov Tue Apr 3 14:24:56 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 03 Apr 2007 14:24:56 -0500 Subject: [Swift-devel] dot error In-Reply-To: References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <6.0.0.22.2.20070309153112.032e73f0@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> Message-ID: <1175628296.27596.0.camel@blabla.mcs.anl.gov> On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > This graph seems to have the color=lightsalmon attribute appear in the node name > BTW, i have never heard of that color http://www.graphviz.org/Documentation/dotguide.pdf > > Tibi > > On 3/9/07, Tiberiu Stef-Praun wrote: > > Bad dot specification > > > > > > On 3/9/07, Mihael Hategan wrote: > > > If it's a valid .dot specification, dot shouldn't blow. > > > If it's not a valid dot specification, dot should complain, and we > > > should fix swift to produce valid .dot files. > > > > > > We need to figure out which one it is. > > > > > > Mihael > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova wrote: > > > > It is a simple graph: a-b-c-d (where a,b,c are single nodes, and d is 68 > > > > parallel nodes). Could the number of parallel nodes be a problem? > > > > > > > > Nika > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > >No clue. Looks like a pretty big graph. I can somewhat view things, but > > > > >it behaves weirdly. > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova wrote: > > > > > > Thank you! > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And the name eventually > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. Nefedova wrote: > > > > > > > > You'd have to walk me through this. I do not know how to produce the ps > > > > > > > > file out of dot file... > > > > > > > > Sorry! > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > >Identify from ImageMagick quickly eats up all the memory when I try to > > > > > > > > >run it on that file. > > > > > > > > >I'm tempted to conclude that something might be broken with dot. > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. Nefedova wrote: > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot -ograph_big.png -Tpng > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png public_html/ > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, Veronika V. Nefedova wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was unable to produce (or > > > > > > > display) the > > > > > > > > > > > > correct png file out of dot file that was generated by swift > > > > > > > (after the > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > Then I ran the dot command to generate the png file: > > > > > > > > > > > > $dot -ograph_big.png -Tpng swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the browser and got this: > > > > > > > > > > > > The image "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > cannot be > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > -- > > Tiberiu (Tibi) Stef-Praun, PhD > > Research Staff, Computation Institute > > 5640 S. Ellis Ave, #405 > > University of Chicago > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > From nefedova at mcs.anl.gov Tue Apr 3 15:35:31 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 03 Apr 2007 15:35:31 -0500 Subject: [Swift-devel] problem with /gpfs on TG Message-ID: <6.0.0.22.2.20070403152803.03a83d60@mail.mcs.anl.gov> Hi, I have some problems with having my workdir set to /gpfs on Teragrid. When my workdir is set to /home, the workflow works just fine. When I want to run a large run I set my workdir to /gpfs (which has a lot of disk space) and then my workflow stops working. The very beginning of the workflow (that has some small jobs) is working ok, but after the long jobs (66 min each) are finished the system never returns the 'done' status back to swift and thus no new jobs are submitted and the workflow stops. Again, exactly the same workflow works fine when workdir is set to /home. Is it TG's problem or Swift's problem? (unless you tell me its definitely a Swift's problem - I'll send email to TG-support as well). Thanks, Nika From benc at hawaga.org.uk Wed Apr 4 04:37:46 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 4 Apr 2007 09:37:46 +0000 (GMT) Subject: [Swift-devel] Swift Question/Request In-Reply-To: References: Message-ID: On Tue, 3 Apr 2007, Tiberiu Stef-Praun wrote: > I'm thinking that having an example in the user guide would help others as > well. > Maybe we should have a "Patterns in Swift" document. The tutorial doc is probably the place for common pattern examples. -- From nefedova at mcs.anl.gov Wed Apr 4 10:46:55 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Wed, 04 Apr 2007 10:46:55 -0500 Subject: [Swift-devel] weird error Message-ID: <6.0.0.22.2.20070404103636.04e8ca40@mail.mcs.anl.gov> Hi, I came across some weird error. It could be the application's error, or it could be swift's error. Here is what I have. This is my function definition: (file out1) GRAND_FINALE (file in1, file in2, file arr[], string s) { app { fepl "--nosite" "--resultonly" "--wham_outputs" @in1 "--solv_lrc_file" @in2 "--fe_file" @out1; } } It is working just fine. BUT -- I had a typo there: I had 1 extra space in the first 2 arguments. I.e. I had " --nosite" and " --resultonly". These 2 extra spaces completely screwed up the application execution. So I am not sure why -- I would think that it shouldn't really cause any problems. I am Cc Yuqing Deng (its his application that I was running). I would be interested to find out why such a small typo caused such big problems... Thanks, Nika From hategan at mcs.anl.gov Wed Apr 4 11:05:13 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 04 Apr 2007 11:05:13 -0500 Subject: [Swift-devel] weird error In-Reply-To: <6.0.0.22.2.20070404103636.04e8ca40@mail.mcs.anl.gov> References: <6.0.0.22.2.20070404103636.04e8ca40@mail.mcs.anl.gov> Message-ID: <1175702713.19291.0.camel@blabla.mcs.anl.gov> The execution was probably screwed up because the application was receiving incorrect arguments. What was the behavior you expected? Mihael On Wed, 2007-04-04 at 10:46 -0500, Veronika V. Nefedova wrote: > Hi, > > I came across some weird error. It could be the application's error, or it > could be swift's error. Here is what I have. This is my function definition: > > (file out1) GRAND_FINALE (file in1, file in2, file arr[], string s) { > app { > fepl "--nosite" "--resultonly" "--wham_outputs" @in1 "--solv_lrc_file" > @in2 "--fe_file" @out1; > } > } > > It is working just fine. BUT -- I had a typo there: I had 1 extra space in > the first 2 arguments. I.e. I had " --nosite" and " --resultonly". These 2 > extra spaces completely screwed up the application execution. So I am not > sure why -- I would think that it shouldn't really cause any problems. I am > Cc Yuqing Deng (its his application that I was running). > > I would be interested to find out why such a small typo caused such big > problems... > > Thanks, > > Nika > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Wed Apr 4 11:35:22 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 04 Apr 2007 11:35:22 -0500 Subject: [Swift-devel] weird error In-Reply-To: References: <6.0.0.22.2.20070404103636.04e8ca40@mail.mcs.anl.gov> <1175702713.19291.0.camel@blabla.mcs.anl.gov> Message-ID: <1175704522.20444.0.camel@blabla.mcs.anl.gov> Exactly. Swift does not trim arguments. On Wed, 2007-04-04 at 11:32 -0500, Yuqing Deng wrote: > The arguments are parsed in the perl script with > Getopt::Long . If the argument string does not begin with -, > it does not get parsed correctly. It's like doing ls " -C", > which is not equal to ls -C . > > Yuqing > > On 4/4/07, Mihael Hategan wrote: > > The execution was probably screwed up because the application was > > receiving incorrect arguments. What was the behavior you expected? > > > > Mihael > > > > On Wed, 2007-04-04 at 10:46 -0500, Veronika V. Nefedova wrote: > > > Hi, > > > > > > I came across some weird error. It could be the application's error, or it > > > could be swift's error. Here is what I have. This is my function definition: > > > > > > (file out1) GRAND_FINALE (file in1, file in2, file arr[], string s) { > > > app { > > > fepl "--nosite" "--resultonly" "--wham_outputs" @in1 "--solv_lrc_file" > > > @in2 "--fe_file" @out1; > > > } > > > } > > > > > > It is working just fine. BUT -- I had a typo there: I had 1 extra space in > > > the first 2 arguments. I.e. I had " --nosite" and " --resultonly". These 2 > > > extra spaces completely screwed up the application execution. So I am not > > > sure why -- I would think that it shouldn't really cause any problems. I am > > > Cc Yuqing Deng (its his application that I was running). > > > > > > I would be interested to find out why such a small typo caused such big > > > problems... > > > > > > Thanks, > > > > > > Nika > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > From yuqing.deng at gmail.com Wed Apr 4 11:32:04 2007 From: yuqing.deng at gmail.com (Yuqing Deng) Date: Wed, 4 Apr 2007 11:32:04 -0500 Subject: [Swift-devel] weird error In-Reply-To: <1175702713.19291.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070404103636.04e8ca40@mail.mcs.anl.gov> <1175702713.19291.0.camel@blabla.mcs.anl.gov> Message-ID: The arguments are parsed in the perl script with Getopt::Long . If the argument string does not begin with -, it does not get parsed correctly. It's like doing ls " -C", which is not equal to ls -C . Yuqing On 4/4/07, Mihael Hategan wrote: > The execution was probably screwed up because the application was > receiving incorrect arguments. What was the behavior you expected? > > Mihael > > On Wed, 2007-04-04 at 10:46 -0500, Veronika V. Nefedova wrote: > > Hi, > > > > I came across some weird error. It could be the application's error, or it > > could be swift's error. Here is what I have. This is my function definition: > > > > (file out1) GRAND_FINALE (file in1, file in2, file arr[], string s) { > > app { > > fepl "--nosite" "--resultonly" "--wham_outputs" @in1 "--solv_lrc_file" > > @in2 "--fe_file" @out1; > > } > > } > > > > It is working just fine. BUT -- I had a typo there: I had 1 extra space in > > the first 2 arguments. I.e. I had " --nosite" and " --resultonly". These 2 > > extra spaces completely screwed up the application execution. So I am not > > sure why -- I would think that it shouldn't really cause any problems. I am > > Cc Yuqing Deng (its his application that I was running). > > > > I would be interested to find out why such a small typo caused such big > > problems... > > > > Thanks, > > > > Nika > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > From nefedova at mcs.anl.gov Mon Apr 9 11:10:18 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Mon, 09 Apr 2007 11:10:18 -0500 Subject: [Swift-devel] dot error In-Reply-To: <1175628296.27596.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <6.0.0.22.2.20070309153112.032e73f0@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> the problem is -- I still can't display any of my graphs... What can be done? Thanks, Nika At 02:24 PM 4/3/2007, Mihael Hategan wrote: >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > This graph seems to have the color=lightsalmon attribute appear in the > node name > > BTW, i have never heard of that color > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > Tibi > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > Bad dot specification > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > If it's not a valid dot specification, dot should complain, and we > > > > should fix swift to produce valid .dot files. > > > > > > > > We need to figure out which one it is. > > > > > > > > Mihael > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova wrote: > > > > > It is a simple graph: a-b-c-d (where a,b,c are single nodes, and > d is 68 > > > > > parallel nodes). Could the number of parallel nodes be a problem? > > > > > > > > > > Nika > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > >No clue. Looks like a pretty big graph. I can somewhat view > things, but > > > > > >it behaves weirdly. > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova wrote: > > > > > > > Thank you! > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And the name > eventually > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. Nefedova wrote: > > > > > > > > > You'd have to walk me through this. I do not know how to > produce the ps > > > > > > > > > file out of dot file... > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > >Identify from ImageMagick quickly eats up all the memory > when I try to > > > > > > > > > >run it on that file. > > > > > > > > > >I'm tempted to conclude that something might be broken > with dot. > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. Nefedova > wrote: > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot -ograph_big.png -Tpng > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png public_html/ > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, Veronika V. > Nefedova wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was unable to > produce (or > > > > > > > > display) the > > > > > > > > > > > > > correct png file out of dot file that was > generated by swift > > > > > > > > (after the > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > Then I ran the dot command to generate the png file: > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the browser and got > this: > > > > > > > > > > > > > The image > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > cannot be > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > -- > > > Tiberiu (Tibi) Stef-Praun, PhD > > > Research Staff, Computation Institute > > > 5640 S. Ellis Ave, #405 > > > University of Chicago > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > From hategan at mcs.anl.gov Mon Apr 9 11:31:59 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 09 Apr 2007 11:31:59 -0500 Subject: [Swift-devel] dot error In-Reply-To: <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <6.0.0.22.2.20070309153112.032e73f0@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> Message-ID: <1176136320.11253.29.camel@blabla.mcs.anl.gov> The labels should now be quoted. But that was not the problem. Your labels did not have spaces. Dotty displays the file just fine. I don't know what more to say. It's a big graph (the .png is 122715 pixels wide). Most tools are not able to deal with such things. Try scaling down the image. On Mon, 2007-04-09 at 11:10 -0500, Veronika V. Nefedova wrote: > the problem is -- I still can't display any of my graphs... What can be done? > > Thanks, > > Nika > > At 02:24 PM 4/3/2007, Mihael Hategan wrote: > >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > > This graph seems to have the color=lightsalmon attribute appear in the > > node name > > > BTW, i have never heard of that color > > > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > > > > Tibi > > > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > > Bad dot specification > > > > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > > If it's not a valid dot specification, dot should complain, and we > > > > > should fix swift to produce valid .dot files. > > > > > > > > > > We need to figure out which one it is. > > > > > > > > > > Mihael > > > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova wrote: > > > > > > It is a simple graph: a-b-c-d (where a,b,c are single nodes, and > > d is 68 > > > > > > parallel nodes). Could the number of parallel nodes be a problem? > > > > > > > > > > > > Nika > > > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > > >No clue. Looks like a pretty big graph. I can somewhat view > > things, but > > > > > > >it behaves weirdly. > > > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova wrote: > > > > > > > > Thank you! > > > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And the name > > eventually > > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. Nefedova wrote: > > > > > > > > > > You'd have to walk me through this. I do not know how to > > produce the ps > > > > > > > > > > file out of dot file... > > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > > >Identify from ImageMagick quickly eats up all the memory > > when I try to > > > > > > > > > > >run it on that file. > > > > > > > > > > >I'm tempted to conclude that something might be broken > > with dot. > > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. Nefedova > > wrote: > > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot -ograph_big.png -Tpng > > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png public_html/ > > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, Veronika V. > > Nefedova wrote: > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was unable to > > produce (or > > > > > > > > > display) the > > > > > > > > > > > > > > correct png file out of dot file that was > > generated by swift > > > > > > > > > (after the > > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > Then I ran the dot command to generate the png file: > > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the browser and got > > this: > > > > > > > > > > > > > > The image > > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > > cannot be > > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Swift-devel mailing list > > > > > Swift-devel at ci.uchicago.edu > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > -- > > > > Tiberiu (Tibi) Stef-Praun, PhD > > > > Research Staff, Computation Institute > > > > 5640 S. Ellis Ave, #405 > > > > University of Chicago > > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > > > > > > From nefedova at mcs.anl.gov Mon Apr 9 12:37:22 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Mon, 09 Apr 2007 12:37:22 -0500 Subject: [Swift-devel] dot error In-Reply-To: <1176136320.11253.29.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <6.0.0.22.2.20070309153112.032e73f0@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> <1176136320.11253.29.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070409115805.03a79920@mail.mcs.anl.gov> My most recent dot file that I can't display has been produced by vdsk070321 and is called FinalGraph.png: http://www.ci.uchicago.edu/~nefedova/FinalGraph.png I am not sure how can I scale down the image ( I tried several command line options that seem to make no effect on the file's size). It seems that I can't rotate the image from the command line options -- it might be that it could help to display those 68 one-tier jobs in column rather then in the row... Thanks, Nika At 11:31 AM 4/9/2007, Mihael Hategan wrote: >The labels should now be quoted. But that was not the problem. Your >labels did not have spaces. Dotty displays the file just fine. I don't >know what more to say. It's a big graph (the .png is 122715 pixels >wide). Most tools are not able to deal with such things. Try scaling >down the image. > >On Mon, 2007-04-09 at 11:10 -0500, Veronika V. Nefedova wrote: > > the problem is -- I still can't display any of my graphs... What can be > done? > > > > Thanks, > > > > Nika > > > > At 02:24 PM 4/3/2007, Mihael Hategan wrote: > > >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > > > This graph seems to have the color=lightsalmon attribute appear in the > > > node name > > > > BTW, i have never heard of that color > > > > > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > > > > > > > Tibi > > > > > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > > > Bad dot specification > > > > > > > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > > > If it's not a valid dot specification, dot should complain, and we > > > > > > should fix swift to produce valid .dot files. > > > > > > > > > > > > We need to figure out which one it is. > > > > > > > > > > > > Mihael > > > > > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova wrote: > > > > > > > It is a simple graph: a-b-c-d (where a,b,c are single nodes, and > > > d is 68 > > > > > > > parallel nodes). Could the number of parallel nodes be a problem? > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > >No clue. Looks like a pretty big graph. I can somewhat view > > > things, but > > > > > > > >it behaves weirdly. > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova wrote: > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And the name > > > eventually > > > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. Nefedova > wrote: > > > > > > > > > > > You'd have to walk me through this. I do not know how to > > > produce the ps > > > > > > > > > > > file out of dot file... > > > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > > > >Identify from ImageMagick quickly eats up all the > memory > > > when I try to > > > > > > > > > > > >run it on that file. > > > > > > > > > > > >I'm tempted to conclude that something might be broken > > > with dot. > > > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. Nefedova > > > wrote: > > > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot -ograph_big.png -Tpng > > > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png public_html/ > > > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, Veronika V. > > > Nefedova wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was unable to > > > produce (or > > > > > > > > > > display) the > > > > > > > > > > > > > > > correct png file out of dot file that was > > > generated by swift > > > > > > > > > > (after the > > > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > Then I ran the dot command to generate the > png file: > > > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the browser and > got > > > this: > > > > > > > > > > > > > > > The image > > > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > > > cannot be > > > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Swift-devel mailing list > > > > > > Swift-devel at ci.uchicago.edu > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > -- > > > > > Tiberiu (Tibi) Stef-Praun, PhD > > > > > Research Staff, Computation Institute > > > > > 5640 S. Ellis Ave, #405 > > > > > University of Chicago > > > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > > > > > > > > > > > From yongzh at cs.uchicago.edu Mon Apr 9 14:17:22 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Mon, 9 Apr 2007 14:17:22 -0500 (CDT) Subject: [Swift-devel] dot error In-Reply-To: <6.0.0.22.2.20070409115805.03a79920@mail.mcs.anl.gov> References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <6.0.0.22.2.20070309153112.032e73f0@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> <1176136320.11253.29.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409115805.03a79920@mail.mcs.anl.gov> Message-ID: You can actually make the layout from left to right: replace in the dot file: graph [splines="compound", rankdir="TB"]; with graph [splines="compound", rankdir="LR"]; Yong. On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > My most recent dot file that I can't display has been produced by > vdsk070321 and is called FinalGraph.png: > http://www.ci.uchicago.edu/~nefedova/FinalGraph.png > > I am not sure how can I scale down the image ( I tried several command line > options that seem to make no effect on the file's size). It seems that I > can't rotate the image from the command line options -- it might be that it > could help to display those 68 one-tier jobs in column rather then in the > row... > > Thanks, > > Nika > > At 11:31 AM 4/9/2007, Mihael Hategan wrote: > >The labels should now be quoted. But that was not the problem. Your > >labels did not have spaces. Dotty displays the file just fine. I don't > >know what more to say. It's a big graph (the .png is 122715 pixels > >wide). Most tools are not able to deal with such things. Try scaling > >down the image. > > > >On Mon, 2007-04-09 at 11:10 -0500, Veronika V. Nefedova wrote: > > > the problem is -- I still can't display any of my graphs... What can be > > done? > > > > > > Thanks, > > > > > > Nika > > > > > > At 02:24 PM 4/3/2007, Mihael Hategan wrote: > > > >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > > > > This graph seems to have the color=lightsalmon attribute appear in the > > > > node name > > > > > BTW, i have never heard of that color > > > > > > > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > > > > > > > > > > Tibi > > > > > > > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > > > > Bad dot specification > > > > > > > > > > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > > > > If it's not a valid dot specification, dot should complain, and we > > > > > > > should fix swift to produce valid .dot files. > > > > > > > > > > > > > > We need to figure out which one it is. > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova wrote: > > > > > > > > It is a simple graph: a-b-c-d (where a,b,c are single nodes, and > > > > d is 68 > > > > > > > > parallel nodes). Could the number of parallel nodes be a problem? > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > >No clue. Looks like a pretty big graph. I can somewhat view > > > > things, but > > > > > > > > >it behaves weirdly. > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova wrote: > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And the name > > > > eventually > > > > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. Nefedova > > wrote: > > > > > > > > > > > > You'd have to walk me through this. I do not know how to > > > > produce the ps > > > > > > > > > > > > file out of dot file... > > > > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > > > > >Identify from ImageMagick quickly eats up all the > > memory > > > > when I try to > > > > > > > > > > > > >run it on that file. > > > > > > > > > > > > >I'm tempted to conclude that something might be broken > > > > with dot. > > > > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. Nefedova > > > > wrote: > > > > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot -ograph_big.png -Tpng > > > > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png public_html/ > > > > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, Veronika V. > > > > Nefedova wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was unable to > > > > produce (or > > > > > > > > > > > display) the > > > > > > > > > > > > > > > > correct png file out of dot file that was > > > > generated by swift > > > > > > > > > > > (after the > > > > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > Then I ran the dot command to generate the > > png file: > > > > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the browser and > > got > > > > this: > > > > > > > > > > > > > > > > The image > > > > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > > > > cannot be > > > > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Swift-devel mailing list > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Tiberiu (Tibi) Stef-Praun, PhD > > > > > > Research Staff, Computation Institute > > > > > > 5640 S. Ellis Ave, #405 > > > > > > University of Chicago > > > > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From nefedova at mcs.anl.gov Mon Apr 9 16:24:11 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Mon, 09 Apr 2007 16:24:11 -0500 Subject: [Swift-devel] dot error In-Reply-To: References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <6.0.0.22.2.20070309153112.032e73f0@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> <1176136320.11253.29.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409115805.03a79920@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070409162053.03a8d640@mail.mcs.anl.gov> Well - replacing that line made a difference! Now I have something displayed, but definitely not the graph I am looking for: it displays a perfectly black rectangle (?). Any ideas what it is? Thanks, Nika At 02:17 PM 4/9/2007, Yong Zhao wrote: >You can actually make the layout from left to right: > >replace in the dot file: > graph [splines="compound", rankdir="TB"]; >with > graph [splines="compound", rankdir="LR"]; > >Yong. > >On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > My most recent dot file that I can't display has been produced by > > vdsk070321 and is called FinalGraph.png: > > http://www.ci.uchicago.edu/~nefedova/FinalGraph.png > > > > I am not sure how can I scale down the image ( I tried several command line > > options that seem to make no effect on the file's size). It seems that I > > can't rotate the image from the command line options -- it might be that it > > could help to display those 68 one-tier jobs in column rather then in the > > row... > > > > Thanks, > > > > Nika > > > > At 11:31 AM 4/9/2007, Mihael Hategan wrote: > > >The labels should now be quoted. But that was not the problem. Your > > >labels did not have spaces. Dotty displays the file just fine. I don't > > >know what more to say. It's a big graph (the .png is 122715 pixels > > >wide). Most tools are not able to deal with such things. Try scaling > > >down the image. > > > > > >On Mon, 2007-04-09 at 11:10 -0500, Veronika V. Nefedova wrote: > > > > the problem is -- I still can't display any of my graphs... What can be > > > done? > > > > > > > > Thanks, > > > > > > > > Nika > > > > > > > > At 02:24 PM 4/3/2007, Mihael Hategan wrote: > > > > >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > > > > > This graph seems to have the color=lightsalmon attribute appear > in the > > > > > node name > > > > > > BTW, i have never heard of that color > > > > > > > > > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > > > > > > > > > > > > > Tibi > > > > > > > > > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > > > > > Bad dot specification > > > > > > > > > > > > > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > > > > > If it's not a valid dot specification, dot should complain, > and we > > > > > > > > should fix swift to produce valid .dot files. > > > > > > > > > > > > > > > > We need to figure out which one it is. > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova wrote: > > > > > > > > > It is a simple graph: a-b-c-d (where a,b,c are single > nodes, and > > > > > d is 68 > > > > > > > > > parallel nodes). Could the number of parallel nodes be a > problem? > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > >No clue. Looks like a pretty big graph. I can somewhat view > > > > > things, but > > > > > > > > > >it behaves weirdly. > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova > wrote: > > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And > the name > > > > > eventually > > > > > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. Nefedova > > > wrote: > > > > > > > > > > > > > You'd have to walk me through this. I do not know > how to > > > > > produce the ps > > > > > > > > > > > > > file out of dot file... > > > > > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > > > > > >Identify from ImageMagick quickly eats up all the > > > memory > > > > > when I try to > > > > > > > > > > > > > >run it on that file. > > > > > > > > > > > > > >I'm tempted to conclude that something might be > broken > > > > > with dot. > > > > > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. > Nefedova > > > > > wrote: > > > > > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot -ograph_big.png -Tpng > > > > > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png > public_html/ > > > > > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, Veronika V. > > > > > Nefedova wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was unable to > > > > > produce (or > > > > > > > > > > > > display) the > > > > > > > > > > > > > > > > > correct png file out of dot file that was > > > > > generated by swift > > > > > > > > > > > > (after the > > > > > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > Then I ran the dot command to generate the > > > png file: > > > > > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the browser and > > > got > > > > > this: > > > > > > > > > > > > > > > > > The image > > > > > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > > > > > cannot be > > > > > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > Swift-devel mailing list > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Tiberiu (Tibi) Stef-Praun, PhD > > > > > > > Research Staff, Computation Institute > > > > > > > 5640 S. Ellis Ave, #405 > > > > > > > University of Chicago > > > > > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From yongzh at cs.uchicago.edu Mon Apr 9 16:31:22 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Mon, 9 Apr 2007 16:31:22 -0500 (CDT) Subject: [Swift-devel] dot error In-Reply-To: <6.0.0.22.2.20070409162053.03a8d640@mail.mcs.anl.gov> References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <6.0.0.22.2.20070309153112.032e73f0@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> <1176136320.11253.29.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409115805.03a79920@mail.mcs.anl.gov> <6.0.0.22.2.20070409162053.03a8d640@mail.mcs.anl.gov> Message-ID: I tried you finaldot.dot and produced a nice png graph using dot -Tpng -o aa.png finaldot.dot Yong. On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > Well - replacing that line made a difference! Now I have something > displayed, but definitely not the graph I am looking for: it displays a > perfectly black rectangle (?). Any ideas what it is? > > Thanks, > > Nika > > At 02:17 PM 4/9/2007, Yong Zhao wrote: > >You can actually make the layout from left to right: > > > >replace in the dot file: > > graph [splines="compound", rankdir="TB"]; > >with > > graph [splines="compound", rankdir="LR"]; > > > >Yong. > > > >On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > > > My most recent dot file that I can't display has been produced by > > > vdsk070321 and is called FinalGraph.png: > > > http://www.ci.uchicago.edu/~nefedova/FinalGraph.png > > > > > > I am not sure how can I scale down the image ( I tried several command line > > > options that seem to make no effect on the file's size). It seems that I > > > can't rotate the image from the command line options -- it might be that it > > > could help to display those 68 one-tier jobs in column rather then in the > > > row... > > > > > > Thanks, > > > > > > Nika > > > > > > At 11:31 AM 4/9/2007, Mihael Hategan wrote: > > > >The labels should now be quoted. But that was not the problem. Your > > > >labels did not have spaces. Dotty displays the file just fine. I don't > > > >know what more to say. It's a big graph (the .png is 122715 pixels > > > >wide). Most tools are not able to deal with such things. Try scaling > > > >down the image. > > > > > > > >On Mon, 2007-04-09 at 11:10 -0500, Veronika V. Nefedova wrote: > > > > > the problem is -- I still can't display any of my graphs... What can be > > > > done? > > > > > > > > > > Thanks, > > > > > > > > > > Nika > > > > > > > > > > At 02:24 PM 4/3/2007, Mihael Hategan wrote: > > > > > >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > > > > > > This graph seems to have the color=lightsalmon attribute appear > > in the > > > > > > node name > > > > > > > BTW, i have never heard of that color > > > > > > > > > > > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > > > > > > > > > > > > > > > > Tibi > > > > > > > > > > > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > > > > > > Bad dot specification > > > > > > > > > > > > > > > > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > > > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > > > > > > If it's not a valid dot specification, dot should complain, > > and we > > > > > > > > > should fix swift to produce valid .dot files. > > > > > > > > > > > > > > > > > > We need to figure out which one it is. > > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova wrote: > > > > > > > > > > It is a simple graph: a-b-c-d (where a,b,c are single > > nodes, and > > > > > > d is 68 > > > > > > > > > > parallel nodes). Could the number of parallel nodes be a > > problem? > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > >No clue. Looks like a pretty big graph. I can somewhat view > > > > > > things, but > > > > > > > > > > >it behaves weirdly. > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova > > wrote: > > > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And > > the name > > > > > > eventually > > > > > > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. Nefedova > > > > wrote: > > > > > > > > > > > > > > You'd have to walk me through this. I do not know > > how to > > > > > > produce the ps > > > > > > > > > > > > > > file out of dot file... > > > > > > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > > > > > > >Identify from ImageMagick quickly eats up all the > > > > memory > > > > > > when I try to > > > > > > > > > > > > > > >run it on that file. > > > > > > > > > > > > > > >I'm tempted to conclude that something might be > > broken > > > > > > with dot. > > > > > > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. > > Nefedova > > > > > > wrote: > > > > > > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot -ograph_big.png -Tpng > > > > > > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png > > public_html/ > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, Veronika V. > > > > > > Nefedova wrote: > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was unable to > > > > > > produce (or > > > > > > > > > > > > > display) the > > > > > > > > > > > > > > > > > > correct png file out of dot file that was > > > > > > generated by swift > > > > > > > > > > > > > (after the > > > > > > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > Then I ran the dot command to generate the > > > > png file: > > > > > > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the browser and > > > > got > > > > > > this: > > > > > > > > > > > > > > > > > > The image > > > > > > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > > > > > > cannot be > > > > > > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > Swift-devel mailing list > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Tiberiu (Tibi) Stef-Praun, PhD > > > > > > > > Research Staff, Computation Institute > > > > > > > > 5640 S. Ellis Ave, #405 > > > > > > > > University of Chicago > > > > > > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > From nefedova at mcs.anl.gov Mon Apr 9 16:40:01 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Mon, 09 Apr 2007 16:40:01 -0500 Subject: [Swift-devel] dot error In-Reply-To: References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <6.0.0.22.2.20070309153112.032e73f0@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> <1176136320.11253.29.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409115805.03a79920@mail.mcs.anl.gov> <6.0.0.22.2.20070409162053.03a8d640@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070409163657.051b0be0@mail.mcs.anl.gov> Hmmm.... This is the command I used to produce the png: dot -oFinalGraph.png -Tpng finaldot.dot Interesting. Nika At 04:31 PM 4/9/2007, Yong Zhao wrote: >I tried you finaldot.dot and produced a nice png graph using >dot -Tpng -o aa.png finaldot.dot > >Yong. > >On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > Well - replacing that line made a difference! Now I have something > > displayed, but definitely not the graph I am looking for: it displays a > > perfectly black rectangle (?). Any ideas what it is? > > > > Thanks, > > > > Nika > > > > At 02:17 PM 4/9/2007, Yong Zhao wrote: > > >You can actually make the layout from left to right: > > > > > >replace in the dot file: > > > graph [splines="compound", rankdir="TB"]; > > >with > > > graph [splines="compound", rankdir="LR"]; > > > > > >Yong. > > > > > >On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > My most recent dot file that I can't display has been produced by > > > > vdsk070321 and is called FinalGraph.png: > > > > http://www.ci.uchicago.edu/~nefedova/FinalGraph.png > > > > > > > > I am not sure how can I scale down the image ( I tried several > command line > > > > options that seem to make no effect on the file's size). It seems > that I > > > > can't rotate the image from the command line options -- it might be > that it > > > > could help to display those 68 one-tier jobs in column rather then > in the > > > > row... > > > > > > > > Thanks, > > > > > > > > Nika > > > > > > > > At 11:31 AM 4/9/2007, Mihael Hategan wrote: > > > > >The labels should now be quoted. But that was not the problem. Your > > > > >labels did not have spaces. Dotty displays the file just fine. I don't > > > > >know what more to say. It's a big graph (the .png is 122715 pixels > > > > >wide). Most tools are not able to deal with such things. Try scaling > > > > >down the image. > > > > > > > > > >On Mon, 2007-04-09 at 11:10 -0500, Veronika V. Nefedova wrote: > > > > > > the problem is -- I still can't display any of my graphs... > What can be > > > > > done? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Nika > > > > > > > > > > > > At 02:24 PM 4/3/2007, Mihael Hategan wrote: > > > > > > >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > > > > > > > This graph seems to have the color=lightsalmon attribute appear > > > in the > > > > > > > node name > > > > > > > > BTW, i have never heard of that color > > > > > > > > > > > > > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > > > > > > > > > > > > > > > > > > > Tibi > > > > > > > > > > > > > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > > > > > > > Bad dot specification > > > > > > > > > > > > > > > > > > > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > > > > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > > > > > > > If it's not a valid dot specification, dot should complain, > > > and we > > > > > > > > > > should fix swift to produce valid .dot files. > > > > > > > > > > > > > > > > > > > > We need to figure out which one it is. > > > > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova > wrote: > > > > > > > > > > > It is a simple graph: a-b-c-d (where a,b,c are single > > > nodes, and > > > > > > > d is 68 > > > > > > > > > > > parallel nodes). Could the number of parallel nodes be a > > > problem? > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > >No clue. Looks like a pretty big graph. I can > somewhat view > > > > > > > things, but > > > > > > > > > > > >it behaves weirdly. > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova > > > wrote: > > > > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And > > > the name > > > > > > > eventually > > > > > > > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. > Nefedova > > > > > wrote: > > > > > > > > > > > > > > > You'd have to walk me through this. I do not know > > > how to > > > > > > > produce the ps > > > > > > > > > > > > > > > file out of dot file... > > > > > > > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > > > > > > > >Identify from ImageMagick quickly eats up > all the > > > > > memory > > > > > > > when I try to > > > > > > > > > > > > > > > >run it on that file. > > > > > > > > > > > > > > > >I'm tempted to conclude that something might be > > > broken > > > > > > > with dot. > > > > > > > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. > > > Nefedova > > > > > > > wrote: > > > > > > > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot > -ograph_big.png -Tpng > > > > > > > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png > > > public_html/ > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, > Veronika V. > > > > > > > Nefedova wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was > unable to > > > > > > > produce (or > > > > > > > > > > > > > > display) the > > > > > > > > > > > > > > > > > > > correct png file out of dot file that was > > > > > > > generated by swift > > > > > > > > > > > > > > (after the > > > > > > > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > Then I ran the dot command to > generate the > > > > > png file: > > > > > > > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the > browser and > > > > > got > > > > > > > this: > > > > > > > > > > > > > > > > > > > The image > > > > > > > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > > > > > > > cannot be > > > > > > > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Tiberiu (Tibi) Stef-Praun, PhD > > > > > > > > > Research Staff, Computation Institute > > > > > > > > > 5640 S. Ellis Ave, #405 > > > > > > > > > University of Chicago > > > > > > > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > From hategan at mcs.anl.gov Mon Apr 9 16:47:10 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 09 Apr 2007 16:47:10 -0500 Subject: [Swift-devel] Re: MolDyn workflow graph In-Reply-To: References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> <1176136320.11253.29.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409115805.03a79920@mail.mcs.anl.gov> <6.0.0.22.2.20070409162053.03a8d640@mail.mcs.anl.gov> <6.0.0.22.2.20070409163657.051b0be0@mail.mcs.anl.gov> Message-ID: <1176155230.22712.0.camel@blabla.mcs.anl.gov> There's this trick to generate a PS file and rasterize it to png with anti-aliasing enabled. On Mon, 2007-04-09 at 16:44 -0500, Yong Zhao wrote: > I tweaked the dot file a bit to compress the graph, attached is the > nice "mushroom" graph. > > Yong. > > On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > Hmmm.... This is the command I used to produce the png: > > dot -oFinalGraph.png -Tpng finaldot.dot > > > > Interesting. > > > > Nika > > > > At 04:31 PM 4/9/2007, Yong Zhao wrote: > > >I tried you finaldot.dot and produced a nice png graph using > > >dot -Tpng -o aa.png finaldot.dot > > > > > >Yong. > > > > > >On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > Well - replacing that line made a difference! Now I have something > > > > displayed, but definitely not the graph I am looking for: it displays a > > > > perfectly black rectangle (?). Any ideas what it is? > > > > > > > > Thanks, > > > > > > > > Nika > > > > > > > > At 02:17 PM 4/9/2007, Yong Zhao wrote: > > > > >You can actually make the layout from left to right: > > > > > > > > > >replace in the dot file: > > > > > graph [splines="compound", rankdir="TB"]; > > > > >with > > > > > graph [splines="compound", rankdir="LR"]; > > > > > > > > > >Yong. > > > > > > > > > >On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > > > My most recent dot file that I can't display has been produced by > > > > > > vdsk070321 and is called FinalGraph.png: > > > > > > http://www.ci.uchicago.edu/~nefedova/FinalGraph.png > > > > > > > > > > > > I am not sure how can I scale down the image ( I tried several > > > command line > > > > > > options that seem to make no effect on the file's size). It seems > > > that I > > > > > > can't rotate the image from the command line options -- it might be > > > that it > > > > > > could help to display those 68 one-tier jobs in column rather then > > > in the > > > > > > row... > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Nika > > > > > > > > > > > > At 11:31 AM 4/9/2007, Mihael Hategan wrote: > > > > > > >The labels should now be quoted. But that was not the problem. Your > > > > > > >labels did not have spaces. Dotty displays the file just fine. I don't > > > > > > >know what more to say. It's a big graph (the .png is 122715 pixels > > > > > > >wide). Most tools are not able to deal with such things. Try scaling > > > > > > >down the image. > > > > > > > > > > > > > >On Mon, 2007-04-09 at 11:10 -0500, Veronika V. Nefedova wrote: > > > > > > > > the problem is -- I still can't display any of my graphs... > > > What can be > > > > > > > done? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > At 02:24 PM 4/3/2007, Mihael Hategan wrote: > > > > > > > > >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > > > > > > > > > This graph seems to have the color=lightsalmon attribute appear > > > > > in the > > > > > > > > > node name > > > > > > > > > > BTW, i have never heard of that color > > > > > > > > > > > > > > > > > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tibi > > > > > > > > > > > > > > > > > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > > > > > > > > > Bad dot specification > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > > > > > > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > > > > > > > > > If it's not a valid dot specification, dot should complain, > > > > > and we > > > > > > > > > > > > should fix swift to produce valid .dot files. > > > > > > > > > > > > > > > > > > > > > > > > We need to figure out which one it is. > > > > > > > > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova > > > wrote: > > > > > > > > > > > > > It is a simple graph: a-b-c-d (where a,b,c are single > > > > > nodes, and > > > > > > > > > d is 68 > > > > > > > > > > > > > parallel nodes). Could the number of parallel nodes be a > > > > > problem? > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > > >No clue. Looks like a pretty big graph. I can > > > somewhat view > > > > > > > > > things, but > > > > > > > > > > > > > >it behaves weirdly. > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova > > > > > wrote: > > > > > > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > > > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And > > > > > the name > > > > > > > > > eventually > > > > > > > > > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. > > > Nefedova > > > > > > > wrote: > > > > > > > > > > > > > > > > > You'd have to walk me through this. I do not know > > > > > how to > > > > > > > > > produce the ps > > > > > > > > > > > > > > > > > file out of dot file... > > > > > > > > > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > > > > > > > > > >Identify from ImageMagick quickly eats up > > > all the > > > > > > > memory > > > > > > > > > when I try to > > > > > > > > > > > > > > > > > >run it on that file. > > > > > > > > > > > > > > > > > >I'm tempted to conclude that something might be > > > > > broken > > > > > > > > > with dot. > > > > > > > > > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. > > > > > Nefedova > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot > > > -ograph_big.png -Tpng > > > > > > > > > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png > > > > > public_html/ > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, > > > Veronika V. > > > > > > > > > Nefedova wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was > > > unable to > > > > > > > > > produce (or > > > > > > > > > > > > > > > > display) the > > > > > > > > > > > > > > > > > > > > > correct png file out of dot file that was > > > > > > > > > generated by swift > > > > > > > > > > > > > > > > (after the > > > > > > > > > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > Then I ran the dot command to > > > generate the > > > > > > > png file: > > > > > > > > > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the > > > browser and > > > > > > > got > > > > > > > > > this: > > > > > > > > > > > > > > > > > > > > > The image > > > > > > > > > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > > > > > > > > > cannot be > > > > > > > > > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Tiberiu (Tibi) Stef-Praun, PhD > > > > > > > > > > > Research Staff, Computation Institute > > > > > > > > > > > 5640 S. Ellis Ave, #405 > > > > > > > > > > > University of Chicago > > > > > > > > > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Swift-devel mailing list > > > > > > Swift-devel at ci.uchicago.edu > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > From yongzh at cs.uchicago.edu Mon Apr 9 16:44:41 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Mon, 9 Apr 2007 16:44:41 -0500 (CDT) Subject: [Swift-devel] MolDyn workflow graph In-Reply-To: <6.0.0.22.2.20070409163657.051b0be0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070309145104.048c5010@mail.mcs.anl.gov> <1173475865.13614.6.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309153648.0553a930@mail.mcs.anl.gov> <1173479931.15803.8.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309165000.032ff910@mail.mcs.anl.gov> <1173484840.19054.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070309180625.05412d40@mail.mcs.anl.gov> <1173492320.19899.1.camel@blabla.mcs.anl.gov> <1175628296.27596.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409110929.052618a0@mail.mcs.anl.gov> <1176136320.11253.29.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070409115805.03a79920@mail.mcs.anl.gov> <6.0.0.22.2.20070409162053.03a8d640@mail.mcs.anl.gov> <6.0.0.22.2.20070409163657.051b0be0@mail.mcs.anl.gov> Message-ID: I tweaked the dot file a bit to compress the graph, attached is the nice "mushroom" graph. Yong. On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > Hmmm.... This is the command I used to produce the png: > dot -oFinalGraph.png -Tpng finaldot.dot > > Interesting. > > Nika > > At 04:31 PM 4/9/2007, Yong Zhao wrote: > >I tried you finaldot.dot and produced a nice png graph using > >dot -Tpng -o aa.png finaldot.dot > > > >Yong. > > > >On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > > > Well - replacing that line made a difference! Now I have something > > > displayed, but definitely not the graph I am looking for: it displays a > > > perfectly black rectangle (?). Any ideas what it is? > > > > > > Thanks, > > > > > > Nika > > > > > > At 02:17 PM 4/9/2007, Yong Zhao wrote: > > > >You can actually make the layout from left to right: > > > > > > > >replace in the dot file: > > > > graph [splines="compound", rankdir="TB"]; > > > >with > > > > graph [splines="compound", rankdir="LR"]; > > > > > > > >Yong. > > > > > > > >On Mon, 9 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > My most recent dot file that I can't display has been produced by > > > > > vdsk070321 and is called FinalGraph.png: > > > > > http://www.ci.uchicago.edu/~nefedova/FinalGraph.png > > > > > > > > > > I am not sure how can I scale down the image ( I tried several > > command line > > > > > options that seem to make no effect on the file's size). It seems > > that I > > > > > can't rotate the image from the command line options -- it might be > > that it > > > > > could help to display those 68 one-tier jobs in column rather then > > in the > > > > > row... > > > > > > > > > > Thanks, > > > > > > > > > > Nika > > > > > > > > > > At 11:31 AM 4/9/2007, Mihael Hategan wrote: > > > > > >The labels should now be quoted. But that was not the problem. Your > > > > > >labels did not have spaces. Dotty displays the file just fine. I don't > > > > > >know what more to say. It's a big graph (the .png is 122715 pixels > > > > > >wide). Most tools are not able to deal with such things. Try scaling > > > > > >down the image. > > > > > > > > > > > >On Mon, 2007-04-09 at 11:10 -0500, Veronika V. Nefedova wrote: > > > > > > > the problem is -- I still can't display any of my graphs... > > What can be > > > > > > done? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > At 02:24 PM 4/3/2007, Mihael Hategan wrote: > > > > > > > >On Tue, 2007-04-03 at 13:09 -0500, Tiberiu Stef-Praun wrote: > > > > > > > > > This graph seems to have the color=lightsalmon attribute appear > > > > in the > > > > > > > > node name > > > > > > > > > BTW, i have never heard of that color > > > > > > > > > > > > > > > >http://www.graphviz.org/Documentation/dotguide.pdf > > > > > > > > > > > > > > > > > > > > > > > > > > Tibi > > > > > > > > > > > > > > > > > > On 3/9/07, Tiberiu Stef-Praun wrote: > > > > > > > > > > Bad dot specification > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 3/9/07, Mihael Hategan wrote: > > > > > > > > > > > If it's a valid .dot specification, dot shouldn't blow. > > > > > > > > > > > If it's not a valid dot specification, dot should complain, > > > > and we > > > > > > > > > > > should fix swift to produce valid .dot files. > > > > > > > > > > > > > > > > > > > > > > We need to figure out which one it is. > > > > > > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > > > > > > > > > On Fri, 2007-03-09 at 18:08 -0600, Veronika V. Nefedova > > wrote: > > > > > > > > > > > > It is a simple graph: a-b-c-d (where a,b,c are single > > > > nodes, and > > > > > > > > d is 68 > > > > > > > > > > > > parallel nodes). Could the number of parallel nodes be a > > > > problem? > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > At 06:00 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > >No clue. Looks like a pretty big graph. I can > > somewhat view > > > > > > > > things, but > > > > > > > > > > > > >it behaves weirdly. > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 16:50 -0600, Veronika V. Nefedova > > > > wrote: > > > > > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Done. Its in my home dir on evitable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 04:38 PM 3/9/2007, you wrote: > > > > > > > > > > > > > > >No walk. Only a step. Instead of -Tpng, -Tps. And > > > > the name > > > > > > > > eventually > > > > > > > > > > > > > > >changed to graph_big.ps. Ok, it was two steps. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:37 -0600, Veronika V. > > Nefedova > > > > > > wrote: > > > > > > > > > > > > > > > > You'd have to walk me through this. I do not know > > > > how to > > > > > > > > produce the ps > > > > > > > > > > > > > > > > file out of dot file... > > > > > > > > > > > > > > > > Sorry! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:31 PM 3/9/2007, you wrote: > > > > > > > > > > > > > > > > >Identify from ImageMagick quickly eats up > > all the > > > > > > memory > > > > > > > > when I try to > > > > > > > > > > > > > > > > >run it on that file. > > > > > > > > > > > > > > > > >I'm tempted to conclude that something might be > > > > broken > > > > > > > > with dot. > > > > > > > > > > > > > > > > >Can you try producing a PostScript file instead? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Mihael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 15:31 -0600, Veronika V. > > > > Nefedova > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Nope! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ dot > > -ograph_big.png -Tpng > > > > > > > > > > > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ cp graph_big.png > > > > public_html/ > > > > > > > > > > > > > > > > > > [nefedova at evitable ~]$ which dot > > > > > > > > > > > > > > > > > > /usr/bin/dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 03:25 PM 3/9/2007, Mihael Hategan wrote: > > > > > > > > > > > > > > > > > > >Did dot complain about anything? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, 2007-03-09 at 14:55 -0600, > > Veronika V. > > > > > > > > Nefedova wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not sure what happened -- but I was > > unable to > > > > > > > > produce (or > > > > > > > > > > > > > > > display) the > > > > > > > > > > > > > > > > > > > > correct png file out of dot file that was > > > > > > > > generated by swift > > > > > > > > > > > > > > > (after the > > > > > > > > > > > > > > > > > > > > workflow was done). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I put my dot file on evitable in > > > > > > > > > > > > > > > > > > > > > ~nefedova/swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > Then I ran the dot command to > > generate the > > > > > > png file: > > > > > > > > > > > > > > > > > > > > $dot -ograph_big.png -Tpng > > > > > > > > swift-MolDyn-free-01dqgqzbhns11.dot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then I tried to display it in the > > browser and > > > > > > got > > > > > > > > this: > > > > > > > > > > > > > > > > > > > > The image > > > > > > > > "http://www.ci.uchicago.edu/~nefedova/graph_big.png" > > > > > > > > > > > > > > > > > cannot be > > > > > > > > > > > > > > > > > > > > displayed, because it contains errors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea what is wrong here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Tiberiu (Tibi) Stef-Praun, PhD > > > > > > > > > > Research Staff, Computation Institute > > > > > > > > > > 5640 S. Ellis Ave, #405 > > > > > > > > > > University of Chicago > > > > > > > > > > http://www-unix.mcs.anl.gov/~tiberius/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Swift-devel mailing list > > > > > Swift-devel at ci.uchicago.edu > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: MolDyn.png Type: image/png Size: 35932 bytes Desc: URL: From benc at hawaga.org.uk Tue Apr 10 00:18:40 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 10 Apr 2007 12:18:40 +0700 (ICT) Subject: [Swift-devel] sanity check on expected behaviour Message-ID: I have the following. $ cat b.swift type file {}; p(file i) { print(@filename(i)); } file q <"volume.blah">; p(q); $ swift b.swift Swift V 0.0405 RunID: 7kplmgxingky0 q I was expecting the printed outptu to be "volume.blah" rather than "q". It looks like @functions that are used as parameters to a procedure (rather than as parameters to a mapper inside a block) don't get passed through properly in the .xml to .kml translation - instead it looks like the function is ignored and the parameter passed on, so that this definition of p() compiles to exactly the same .kml code as the one above: p(file i) { print(i); } I guess either the compiler should warn of this rather than continuing silenty, or (more preferably, I think) @functions should actually work in this context. -- From tiberius at ci.uchicago.edu Tue Apr 10 11:47:51 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Tue, 10 Apr 2007 11:47:51 -0500 Subject: [Swift-devel] Interesting observation when running Swift Message-ID: I have a workflow along these lines: // this one generates outputsStage1[] stage1() // this one merges the stage1 outputs stage2(outputsStage1[]) note that it is not outputsStage1=stage1() Since the outputsStage1 files were not generated yet, I expected that Karajan waited for them to be created before running stage2, but that was not the case: stage2 was executed when the workflow started (and it failed) and caused the workflow to fail. I know how to fix the workflow, that is not the issue. The issue is that I expected the workflow to be data-driven, but it seems to be code driven. Explanation: it attempted to execute a section even if its input files were not available. Correct me if I am wrong. Tibi -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From hategan at mcs.anl.gov Tue Apr 10 11:50:55 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Apr 2007 11:50:55 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: References: Message-ID: <1176223855.10019.2.camel@blabla.mcs.anl.gov> Swift doesn't monitor the file system. Data driven doesn't mean that it does magic in the background. It means that you have to express data dependencies in the code. On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > I have a workflow along these lines: > > // this one generates outputsStage1[] > stage1() > // this one merges the stage1 outputs > stage2(outputsStage1[]) > > note that it is not outputsStage1=stage1() > > Since the outputsStage1 files were not generated yet, I expected that > Karajan waited for them to be created before running stage2, but that > was not the case: stage2 was executed when the workflow started (and > it failed) and caused the workflow to fail. > > I know how to fix the workflow, that is not the issue. The issue is > that I expected the workflow to be data-driven, but it seems to be > code driven. Explanation: it attempted to execute a section even if > its input files were not available. > > Correct me if I am wrong. > Tibi > From tiberius at ci.uchicago.edu Tue Apr 10 11:57:04 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Tue, 10 Apr 2007 11:57:04 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <1176223855.10019.2.camel@blabla.mcs.anl.gov> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> Message-ID: Interesting. Does anyone else think that monitoring the filesystem could be a useful idea ? For instance it could help with file-driven dependencies, in scenarios where we want to have continuous workflows, or compose independent wokflows. The filesystem would act as the publish-subscribe mechanism for some workflow cases. Tibi On 4/10/07, Mihael Hategan wrote: > Swift doesn't monitor the file system. > Data driven doesn't mean that it does magic in the background. It means > that you have to express data dependencies in the code. > > On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > > I have a workflow along these lines: > > > > // this one generates outputsStage1[] > > stage1() > > // this one merges the stage1 outputs > > stage2(outputsStage1[]) > > > > note that it is not outputsStage1=stage1() > > > > Since the outputsStage1 files were not generated yet, I expected that > > Karajan waited for them to be created before running stage2, but that > > was not the case: stage2 was executed when the workflow started (and > > it failed) and caused the workflow to fail. > > > > I know how to fix the workflow, that is not the issue. The issue is > > that I expected the workflow to be data-driven, but it seems to be > > code driven. Explanation: it attempted to execute a section even if > > its input files were not available. > > > > Correct me if I am wrong. > > Tibi > > > > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From nefedova at mcs.anl.gov Tue Apr 10 12:12:12 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 10 Apr 2007 12:12:12 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> I think that something like that would be useful: outputsStage1[]=stage1() outputsStage2[]=stage2(outputsStage1[]) if you didn't have to specify the number or specific filenames for the outputs. Basically it would be good for the Workflow engine to understand this: "get all the produced files from stage 1 and use them as an input for Stage 2" (; Nika At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: >Interesting. > >Does anyone else think that monitoring the filesystem could be a useful idea ? > >For instance it could help with file-driven dependencies, in scenarios >where we want to have continuous workflows, or compose independent >wokflows. The filesystem would act as the publish-subscribe mechanism >for some workflow cases. > >Tibi > >On 4/10/07, Mihael Hategan wrote: >>Swift doesn't monitor the file system. >>Data driven doesn't mean that it does magic in the background. It means >>that you have to express data dependencies in the code. >> >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: >> > I have a workflow along these lines: >> > >> > // this one generates outputsStage1[] >> > stage1() >> > // this one merges the stage1 outputs >> > stage2(outputsStage1[]) >> > >> > note that it is not outputsStage1=stage1() >> > >> > Since the outputsStage1 files were not generated yet, I expected that >> > Karajan waited for them to be created before running stage2, but that >> > was not the case: stage2 was executed when the workflow started (and >> > it failed) and caused the workflow to fail. >> > >> > I know how to fix the workflow, that is not the issue. The issue is >> > that I expected the workflow to be data-driven, but it seems to be >> > code driven. Explanation: it attempted to execute a section even if >> > its input files were not available. >> > >> > Correct me if I am wrong. >> > Tibi >> > >> > > >-- >Tiberiu (Tibi) Stef-Praun, PhD >Research Staff, Computation Institute >5640 S. Ellis Ave, #405 >University of Chicago >http://www-unix.mcs.anl.gov/~tiberius/ >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Tue Apr 10 12:20:06 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Apr 2007 12:20:06 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> Message-ID: <1176225606.10451.21.camel@blabla.mcs.anl.gov> That's the VDC scenario to a large extent, where the VDC is a filesystem with metadata and a few nice things like locking, replication, and listeners (publish/subscribe). In principle, this is what the futures +Swift types are: an in-memory file system with a p/s mechanism. Efficient implementation of such things on top of existing filesystems is, I think, not that easy. Note that the p/s mechanism would need to be emulated (perhaps by polling the disks). Location-independence may also prove tricky, since you'd need to support these for things like GridFTP. So yes, it would be useful, but pretty difficult, and then you start asking yourself if another solution wouldn't achieve the same thing faster. On Tue, 2007-04-10 at 11:57 -0500, Tiberiu Stef-Praun wrote: > Interesting. > > Does anyone else think that monitoring the filesystem could be a useful idea ? > > For instance it could help with file-driven dependencies, in scenarios > where we want to have continuous workflows, or compose independent > wokflows. The filesystem would act as the publish-subscribe mechanism > for some workflow cases. > > Tibi > > On 4/10/07, Mihael Hategan wrote: > > Swift doesn't monitor the file system. > > Data driven doesn't mean that it does magic in the background. It means > > that you have to express data dependencies in the code. > > > > On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > > > I have a workflow along these lines: > > > > > > // this one generates outputsStage1[] > > > stage1() > > > // this one merges the stage1 outputs > > > stage2(outputsStage1[]) > > > > > > note that it is not outputsStage1=stage1() > > > > > > Since the outputsStage1 files were not generated yet, I expected that > > > Karajan waited for them to be created before running stage2, but that > > > was not the case: stage2 was executed when the workflow started (and > > > it failed) and caused the workflow to fail. > > > > > > I know how to fix the workflow, that is not the issue. The issue is > > > that I expected the workflow to be data-driven, but it seems to be > > > code driven. Explanation: it attempted to execute a section even if > > > its input files were not available. > > > > > > Correct me if I am wrong. > > > Tibi > > > > > > > > > From hategan at mcs.anl.gov Tue Apr 10 12:23:06 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Apr 2007 12:23:06 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> Message-ID: <1176225786.10451.25.camel@blabla.mcs.anl.gov> On Tue, 2007-04-10 at 12:12 -0500, Veronika V. Nefedova wrote: > I think that something like that would be useful: > > outputsStage1[]=stage1() > outputsStage2[]=stage2(outputsStage1[]) > > if you didn't have to specify the number or specific filenames for the > outputs. Basically it would be good for the Workflow engine to understand > this: "get all the produced files from stage 1 and use them as an input for > Stage 2" Some applications are known to produce extra temporary files. Mappers are supposed to be able to extract arrays from a cluttered file system (assuming that there are no ambiguities in naming patterns). So yes, it would be useful, and I think one of the planned features, and I see no obvious problems besides the possible naming conflicts (which would apply to a local file system anyway, so not a new problem). > > (; > > Nika > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > >Interesting. > > > >Does anyone else think that monitoring the filesystem could be a useful idea ? > > > >For instance it could help with file-driven dependencies, in scenarios > >where we want to have continuous workflows, or compose independent > >wokflows. The filesystem would act as the publish-subscribe mechanism > >for some workflow cases. > > > >Tibi > > > >On 4/10/07, Mihael Hategan wrote: > >>Swift doesn't monitor the file system. > >>Data driven doesn't mean that it does magic in the background. It means > >>that you have to express data dependencies in the code. > >> > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > >> > I have a workflow along these lines: > >> > > >> > // this one generates outputsStage1[] > >> > stage1() > >> > // this one merges the stage1 outputs > >> > stage2(outputsStage1[]) > >> > > >> > note that it is not outputsStage1=stage1() > >> > > >> > Since the outputsStage1 files were not generated yet, I expected that > >> > Karajan waited for them to be created before running stage2, but that > >> > was not the case: stage2 was executed when the workflow started (and > >> > it failed) and caused the workflow to fail. > >> > > >> > I know how to fix the workflow, that is not the issue. The issue is > >> > that I expected the workflow to be data-driven, but it seems to be > >> > code driven. Explanation: it attempted to execute a section even if > >> > its input files were not available. > >> > > >> > Correct me if I am wrong. > >> > Tibi > >> > > >> > > > > > >-- > >Tiberiu (Tibi) Stef-Praun, PhD > >Research Staff, Computation Institute > >5640 S. Ellis Ave, #405 > >University of Chicago > >http://www-unix.mcs.anl.gov/~tiberius/ > >_______________________________________________ > >Swift-devel mailing list > >Swift-devel at ci.uchicago.edu > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From itf at mcs.anl.gov Tue Apr 10 12:46:08 2007 From: itf at mcs.anl.gov (=?UTF-8?B?SWFuIEZvc3Rlcg==?=) Date: Tue, 10 Apr 2007 17:46:08 +0000 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> Message-ID: <1195630042-1176227343-cardhu_blackberry.rim.net-1925209672-@bwe053-cell00.bisx.prod.on.blackberry> I thought we could already do that. We can, I think, in the case of the first stage--we can take all files in a directory as a datset, say? But we can't do that for latter stages? Sent via BlackBerry from T-Mobile -----Original Message----- From: "Veronika V. Nefedova" Date: Tue, 10 Apr 2007 12:12:12 To:"Tiberiu Stef-Praun" , "Mihael Hategan" Cc:swift-devel at ci.uchicago.edu Subject: Re: [Swift-devel] Interesting observation when running Swift I think that something like that would be useful: outputsStage1[]=stage1() outputsStage2[]=stage2(outputsStage1[]) if you didn't have to specify the number or specific filenames for the outputs. Basically it would be good for the Workflow engine to understand this: "get all the produced files from stage 1 and use them as an input for Stage 2" (; Nika At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: >Interesting. > >Does anyone else think that monitoring the filesystem could be a useful idea ? > >For instance it could help with file-driven dependencies, in scenarios >where we want to have continuous workflows, or compose independent >wokflows. The filesystem would act as the publish-subscribe mechanism >for some workflow cases. > >Tibi > >On 4/10/07, Mihael Hategan wrote: >>Swift doesn't monitor the file system. >>Data driven doesn't mean that it does magic in the background. It means >>that you have to express data dependencies in the code. >> >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: >> > I have a workflow along these lines: >> > >> > // this one generates outputsStage1[] >> > stage1() >> > // this one merges the stage1 outputs >> > stage2(outputsStage1[]) >> > >> > note that it is not outputsStage1=stage1() >> > >> > Since the outputsStage1 files were not generated yet, I expected that >> > Karajan waited for them to be created before running stage2, but that >> > was not the case: stage2 was executed when the workflow started (and >> > it failed) and caused the workflow to fail. >> > >> > I know how to fix the workflow, that is not the issue. The issue is >> > that I expected the workflow to be data-driven, but it seems to be >> > code driven. Explanation: it attempted to execute a section even if >> > its input files were not available. >> > >> > Correct me if I am wrong. >> > Tibi >> > >> > > >-- >Tiberiu (Tibi) Stef-Praun, PhD >Research Staff, Computation Institute >5640 S. Ellis Ave, #405 >University of Chicago >http://www-unix.mcs.anl.gov/~tiberius/ >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From nefedova at mcs.anl.gov Tue Apr 10 12:54:37 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 10 Apr 2007 12:54:37 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <1195630042-1176227343-cardhu_blackberry.rim.net-1925209672 -@bwe053-cell00.bisx.prod.on.blackberry> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1195630042-1176227343-cardhu_blackberry.rim.net-1925209672-@bwe053-cell00.bisx.prod.on.blackberry> Message-ID: <6.0.0.22.2.20070410125322.03a92d30@mail.mcs.anl.gov> The problem is with the output -- you can't produce an unknown (to swift) number of files... Nika At 12:46 PM 4/10/2007, Ian Foster wrote: >I thought we could already do that. We can, I think, in the case of the >first stage--we can take all files in a directory as a datset, say? But we >can't do that for latter stages? > > >Sent via BlackBerry from T-Mobile > >-----Original Message----- >From: "Veronika V. Nefedova" >Date: Tue, 10 Apr 2007 12:12:12 >To:"Tiberiu Stef-Praun" , "Mihael Hategan" > >Cc:swift-devel at ci.uchicago.edu >Subject: Re: [Swift-devel] Interesting observation when running Swift > >I think that something like that would be useful: > >outputsStage1[]=stage1() >outputsStage2[]=stage2(outputsStage1[]) > >if you didn't have to specify the number or specific filenames for the >outputs. Basically it would be good for the Workflow engine to understand >this: "get all the produced files from stage 1 and use them as an input for >Stage 2" > >(; > >Nika > > >At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > >Interesting. > > > >Does anyone else think that monitoring the filesystem could be a useful > idea ? > > > >For instance it could help with file-driven dependencies, in scenarios > >where we want to have continuous workflows, or compose independent > >wokflows. The filesystem would act as the publish-subscribe mechanism > >for some workflow cases. > > > >Tibi > > > >On 4/10/07, Mihael Hategan wrote: > >>Swift doesn't monitor the file system. > >>Data driven doesn't mean that it does magic in the background. It means > >>that you have to express data dependencies in the code. > >> > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > >> > I have a workflow along these lines: > >> > > >> > // this one generates outputsStage1[] > >> > stage1() > >> > // this one merges the stage1 outputs > >> > stage2(outputsStage1[]) > >> > > >> > note that it is not outputsStage1=stage1() > >> > > >> > Since the outputsStage1 files were not generated yet, I expected that > >> > Karajan waited for them to be created before running stage2, but that > >> > was not the case: stage2 was executed when the workflow started (and > >> > it failed) and caused the workflow to fail. > >> > > >> > I know how to fix the workflow, that is not the issue. The issue is > >> > that I expected the workflow to be data-driven, but it seems to be > >> > code driven. Explanation: it attempted to execute a section even if > >> > its input files were not available. > >> > > >> > Correct me if I am wrong. > >> > Tibi > >> > > >> > > > > > >-- > >Tiberiu (Tibi) Stef-Praun, PhD > >Research Staff, Computation Institute > >5640 S. Ellis Ave, #405 > >University of Chicago > >http://www-unix.mcs.anl.gov/~tiberius/ > >_______________________________________________ > >Swift-devel mailing list > >Swift-devel at ci.uchicago.edu > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Tue Apr 10 12:56:55 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Apr 2007 12:56:55 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <1195630042-1176227343-cardhu_blackberry.rim.net-1925209672-@bwe053-cell00.bisx.prod.on.blackberry> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1195630042-1176227343-cardhu_blackberry.rim.net-1925209672-@bwe053-cell00.bisx.prod.on.blackberry> Message-ID: <1176227815.12375.5.camel@blabla.mcs.anl.gov> On Tue, 2007-04-10 at 17:46 +0000, Ian Foster wrote: > I thought we could already do that. We can, I think, in the case of the first stage--we can take all files in a directory as a datset, say? But we can't do that for latter stages? Not much to do with when the stages happen. It's more about knowing what to stage out. Right now mappers can tell what files are relevant in a given collection (directory) that they manage, and the details are left to the implementations of the mappers. What's needed is to feed each mapper for return values from an atomic proc with the list of files generated by an application, let them select the relevant files, then do the stage-out and populate the Swift data structures based on that. > > > Sent via BlackBerry from T-Mobile > > -----Original Message----- > From: "Veronika V. Nefedova" > Date: Tue, 10 Apr 2007 12:12:12 > To:"Tiberiu Stef-Praun" , "Mihael Hategan" > Cc:swift-devel at ci.uchicago.edu > Subject: Re: [Swift-devel] Interesting observation when running Swift > > I think that something like that would be useful: > > outputsStage1[]=stage1() > outputsStage2[]=stage2(outputsStage1[]) > > if you didn't have to specify the number or specific filenames for the > outputs. Basically it would be good for the Workflow engine to understand > this: "get all the produced files from stage 1 and use them as an input for > Stage 2" > > (; > > Nika > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > >Interesting. > > > >Does anyone else think that monitoring the filesystem could be a useful idea ? > > > >For instance it could help with file-driven dependencies, in scenarios > >where we want to have continuous workflows, or compose independent > >wokflows. The filesystem would act as the publish-subscribe mechanism > >for some workflow cases. > > > >Tibi > > > >On 4/10/07, Mihael Hategan wrote: > >>Swift doesn't monitor the file system. > >>Data driven doesn't mean that it does magic in the background. It means > >>that you have to express data dependencies in the code. > >> > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > >> > I have a workflow along these lines: > >> > > >> > // this one generates outputsStage1[] > >> > stage1() > >> > // this one merges the stage1 outputs > >> > stage2(outputsStage1[]) > >> > > >> > note that it is not outputsStage1=stage1() > >> > > >> > Since the outputsStage1 files were not generated yet, I expected that > >> > Karajan waited for them to be created before running stage2, but that > >> > was not the case: stage2 was executed when the workflow started (and > >> > it failed) and caused the workflow to fail. > >> > > >> > I know how to fix the workflow, that is not the issue. The issue is > >> > that I expected the workflow to be data-driven, but it seems to be > >> > code driven. Explanation: it attempted to execute a section even if > >> > its input files were not available. > >> > > >> > Correct me if I am wrong. > >> > Tibi > >> > > >> > > > > > >-- > >Tiberiu (Tibi) Stef-Praun, PhD > >Research Staff, Computation Institute > >5640 S. Ellis Ave, #405 > >University of Chicago > >http://www-unix.mcs.anl.gov/~tiberius/ > >_______________________________________________ > >Swift-devel mailing list > >Swift-devel at ci.uchicago.edu > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From yongzh at cs.uchicago.edu Tue Apr 10 13:01:44 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Tue, 10 Apr 2007 13:01:44 -0500 (CDT) Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <1176225786.10451.25.camel@blabla.mcs.anl.gov> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1176225786.10451.25.camel@blabla.mcs.anl.gov> Message-ID: aren't these already addressed by remote mappers? Yong. On Tue, 10 Apr 2007, Mihael Hategan wrote: > On Tue, 2007-04-10 at 12:12 -0500, Veronika V. Nefedova wrote: > > I think that something like that would be useful: > > > > outputsStage1[]=stage1() > > outputsStage2[]=stage2(outputsStage1[]) > > > > if you didn't have to specify the number or specific filenames for the > > outputs. Basically it would be good for the Workflow engine to understand > > this: "get all the produced files from stage 1 and use them as an input for > > Stage 2" > > Some applications are known to produce extra temporary files. Mappers > are supposed to be able to extract arrays from a cluttered file system > (assuming that there are no ambiguities in naming patterns). So yes, it > would be useful, and I think one of the planned features, and I see no > obvious problems besides the possible naming conflicts (which would > apply to a local file system anyway, so not a new problem). > > > > > (; > > > > Nika > > > > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > > >Interesting. > > > > > >Does anyone else think that monitoring the filesystem could be a useful idea ? > > > > > >For instance it could help with file-driven dependencies, in scenarios > > >where we want to have continuous workflows, or compose independent > > >wokflows. The filesystem would act as the publish-subscribe mechanism > > >for some workflow cases. > > > > > >Tibi > > > > > >On 4/10/07, Mihael Hategan wrote: > > >>Swift doesn't monitor the file system. > > >>Data driven doesn't mean that it does magic in the background. It means > > >>that you have to express data dependencies in the code. > > >> > > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > > >> > I have a workflow along these lines: > > >> > > > >> > // this one generates outputsStage1[] > > >> > stage1() > > >> > // this one merges the stage1 outputs > > >> > stage2(outputsStage1[]) > > >> > > > >> > note that it is not outputsStage1=stage1() > > >> > > > >> > Since the outputsStage1 files were not generated yet, I expected that > > >> > Karajan waited for them to be created before running stage2, but that > > >> > was not the case: stage2 was executed when the workflow started (and > > >> > it failed) and caused the workflow to fail. > > >> > > > >> > I know how to fix the workflow, that is not the issue. The issue is > > >> > that I expected the workflow to be data-driven, but it seems to be > > >> > code driven. Explanation: it attempted to execute a section even if > > >> > its input files were not available. > > >> > > > >> > Correct me if I am wrong. > > >> > Tibi > > >> > > > >> > > > > > > > > >-- > > >Tiberiu (Tibi) Stef-Praun, PhD > > >Research Staff, Computation Institute > > >5640 S. Ellis Ave, #405 > > >University of Chicago > > >http://www-unix.mcs.anl.gov/~tiberius/ > > >_______________________________________________ > > >Swift-devel mailing list > > >Swift-devel at ci.uchicago.edu > > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Tue Apr 10 13:03:31 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Apr 2007 13:03:31 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1176225786.10451.25.camel@blabla.mcs.anl.gov> Message-ID: <1176228211.12875.0.camel@blabla.mcs.anl.gov> Remote mappers? On Tue, 2007-04-10 at 13:01 -0500, Yong Zhao wrote: > aren't these already addressed by remote mappers? > > Yong. > > On Tue, 10 Apr 2007, Mihael Hategan wrote: > > > On Tue, 2007-04-10 at 12:12 -0500, Veronika V. Nefedova wrote: > > > I think that something like that would be useful: > > > > > > outputsStage1[]=stage1() > > > outputsStage2[]=stage2(outputsStage1[]) > > > > > > if you didn't have to specify the number or specific filenames for the > > > outputs. Basically it would be good for the Workflow engine to understand > > > this: "get all the produced files from stage 1 and use them as an input for > > > Stage 2" > > > > Some applications are known to produce extra temporary files. Mappers > > are supposed to be able to extract arrays from a cluttered file system > > (assuming that there are no ambiguities in naming patterns). So yes, it > > would be useful, and I think one of the planned features, and I see no > > obvious problems besides the possible naming conflicts (which would > > apply to a local file system anyway, so not a new problem). > > > > > > > > (; > > > > > > Nika > > > > > > > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > > > >Interesting. > > > > > > > >Does anyone else think that monitoring the filesystem could be a useful idea ? > > > > > > > >For instance it could help with file-driven dependencies, in scenarios > > > >where we want to have continuous workflows, or compose independent > > > >wokflows. The filesystem would act as the publish-subscribe mechanism > > > >for some workflow cases. > > > > > > > >Tibi > > > > > > > >On 4/10/07, Mihael Hategan wrote: > > > >>Swift doesn't monitor the file system. > > > >>Data driven doesn't mean that it does magic in the background. It means > > > >>that you have to express data dependencies in the code. > > > >> > > > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > > > >> > I have a workflow along these lines: > > > >> > > > > >> > // this one generates outputsStage1[] > > > >> > stage1() > > > >> > // this one merges the stage1 outputs > > > >> > stage2(outputsStage1[]) > > > >> > > > > >> > note that it is not outputsStage1=stage1() > > > >> > > > > >> > Since the outputsStage1 files were not generated yet, I expected that > > > >> > Karajan waited for them to be created before running stage2, but that > > > >> > was not the case: stage2 was executed when the workflow started (and > > > >> > it failed) and caused the workflow to fail. > > > >> > > > > >> > I know how to fix the workflow, that is not the issue. The issue is > > > >> > that I expected the workflow to be data-driven, but it seems to be > > > >> > code driven. Explanation: it attempted to execute a section even if > > > >> > its input files were not available. > > > >> > > > > >> > Correct me if I am wrong. > > > >> > Tibi > > > >> > > > > >> > > > > > > > > > > > >-- > > > >Tiberiu (Tibi) Stef-Praun, PhD > > > >Research Staff, Computation Institute > > > >5640 S. Ellis Ave, #405 > > > >University of Chicago > > > >http://www-unix.mcs.anl.gov/~tiberius/ > > > >_______________________________________________ > > > >Swift-devel mailing list > > > >Swift-devel at ci.uchicago.edu > > > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > From nefedova at mcs.anl.gov Tue Apr 10 13:03:43 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 10 Apr 2007 13:03:43 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1176225786.10451.25.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070410130311.03a19350@mail.mcs.anl.gov> do we have the remote mappers available now ? I didn't see any announcement... Nika At 01:01 PM 4/10/2007, Yong Zhao wrote: >aren't these already addressed by remote mappers? > >Yong. > >On Tue, 10 Apr 2007, Mihael Hategan wrote: > > > On Tue, 2007-04-10 at 12:12 -0500, Veronika V. Nefedova wrote: > > > I think that something like that would be useful: > > > > > > outputsStage1[]=stage1() > > > outputsStage2[]=stage2(outputsStage1[]) > > > > > > if you didn't have to specify the number or specific filenames for the > > > outputs. Basically it would be good for the Workflow engine to understand > > > this: "get all the produced files from stage 1 and use them as an > input for > > > Stage 2" > > > > Some applications are known to produce extra temporary files. Mappers > > are supposed to be able to extract arrays from a cluttered file system > > (assuming that there are no ambiguities in naming patterns). So yes, it > > would be useful, and I think one of the planned features, and I see no > > obvious problems besides the possible naming conflicts (which would > > apply to a local file system anyway, so not a new problem). > > > > > > > > (; > > > > > > Nika > > > > > > > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > > > >Interesting. > > > > > > > >Does anyone else think that monitoring the filesystem could be a > useful idea ? > > > > > > > >For instance it could help with file-driven dependencies, in scenarios > > > >where we want to have continuous workflows, or compose independent > > > >wokflows. The filesystem would act as the publish-subscribe mechanism > > > >for some workflow cases. > > > > > > > >Tibi > > > > > > > >On 4/10/07, Mihael Hategan wrote: > > > >>Swift doesn't monitor the file system. > > > >>Data driven doesn't mean that it does magic in the background. It means > > > >>that you have to express data dependencies in the code. > > > >> > > > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > > > >> > I have a workflow along these lines: > > > >> > > > > >> > // this one generates outputsStage1[] > > > >> > stage1() > > > >> > // this one merges the stage1 outputs > > > >> > stage2(outputsStage1[]) > > > >> > > > > >> > note that it is not outputsStage1=stage1() > > > >> > > > > >> > Since the outputsStage1 files were not generated yet, I expected > that > > > >> > Karajan waited for them to be created before running stage2, but > that > > > >> > was not the case: stage2 was executed when the workflow started (and > > > >> > it failed) and caused the workflow to fail. > > > >> > > > > >> > I know how to fix the workflow, that is not the issue. The issue is > > > >> > that I expected the workflow to be data-driven, but it seems to be > > > >> > code driven. Explanation: it attempted to execute a section even if > > > >> > its input files were not available. > > > >> > > > > >> > Correct me if I am wrong. > > > >> > Tibi > > > >> > > > > >> > > > > > > > > > > > >-- > > > >Tiberiu (Tibi) Stef-Praun, PhD > > > >Research Staff, Computation Institute > > > >5640 S. Ellis Ave, #405 > > > >University of Chicago > > > >http://www-unix.mcs.anl.gov/~tiberius/ > > > >_______________________________________________ > > > >Swift-devel mailing list > > > >Swift-devel at ci.uchicago.edu > > > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From yongzh at cs.uchicago.edu Tue Apr 10 13:08:44 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Tue, 10 Apr 2007 13:08:44 -0500 (CDT) Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <1176228211.12875.0.camel@blabla.mcs.anl.gov> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1176225786.10451.25.camel@blabla.mcs.anl.gov> <1176228211.12875.0.camel@blabla.mcs.anl.gov> Message-ID: something Mike has been calling collectors for a long time: which does pattern matching (or other operations) on the remote execution directory and collect the outputs etc. On Tue, 10 Apr 2007, Mihael Hategan wrote: > Remote mappers? > > On Tue, 2007-04-10 at 13:01 -0500, Yong Zhao wrote: > > aren't these already addressed by remote mappers? > > > > Yong. > > > > On Tue, 10 Apr 2007, Mihael Hategan wrote: > > > > > On Tue, 2007-04-10 at 12:12 -0500, Veronika V. Nefedova wrote: > > > > I think that something like that would be useful: > > > > > > > > outputsStage1[]=stage1() > > > > outputsStage2[]=stage2(outputsStage1[]) > > > > > > > > if you didn't have to specify the number or specific filenames for the > > > > outputs. Basically it would be good for the Workflow engine to understand > > > > this: "get all the produced files from stage 1 and use them as an input for > > > > Stage 2" > > > > > > Some applications are known to produce extra temporary files. Mappers > > > are supposed to be able to extract arrays from a cluttered file system > > > (assuming that there are no ambiguities in naming patterns). So yes, it > > > would be useful, and I think one of the planned features, and I see no > > > obvious problems besides the possible naming conflicts (which would > > > apply to a local file system anyway, so not a new problem). > > > > > > > > > > > (; > > > > > > > > Nika > > > > > > > > > > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > > > > >Interesting. > > > > > > > > > >Does anyone else think that monitoring the filesystem could be a useful idea ? > > > > > > > > > >For instance it could help with file-driven dependencies, in scenarios > > > > >where we want to have continuous workflows, or compose independent > > > > >wokflows. The filesystem would act as the publish-subscribe mechanism > > > > >for some workflow cases. > > > > > > > > > >Tibi > > > > > > > > > >On 4/10/07, Mihael Hategan wrote: > > > > >>Swift doesn't monitor the file system. > > > > >>Data driven doesn't mean that it does magic in the background. It means > > > > >>that you have to express data dependencies in the code. > > > > >> > > > > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > > > > >> > I have a workflow along these lines: > > > > >> > > > > > >> > // this one generates outputsStage1[] > > > > >> > stage1() > > > > >> > // this one merges the stage1 outputs > > > > >> > stage2(outputsStage1[]) > > > > >> > > > > > >> > note that it is not outputsStage1=stage1() > > > > >> > > > > > >> > Since the outputsStage1 files were not generated yet, I expected that > > > > >> > Karajan waited for them to be created before running stage2, but that > > > > >> > was not the case: stage2 was executed when the workflow started (and > > > > >> > it failed) and caused the workflow to fail. > > > > >> > > > > > >> > I know how to fix the workflow, that is not the issue. The issue is > > > > >> > that I expected the workflow to be data-driven, but it seems to be > > > > >> > code driven. Explanation: it attempted to execute a section even if > > > > >> > its input files were not available. > > > > >> > > > > > >> > Correct me if I am wrong. > > > > >> > Tibi > > > > >> > > > > > >> > > > > > > > > > > > > > > >-- > > > > >Tiberiu (Tibi) Stef-Praun, PhD > > > > >Research Staff, Computation Institute > > > > >5640 S. Ellis Ave, #405 > > > > >University of Chicago > > > > >http://www-unix.mcs.anl.gov/~tiberius/ > > > > >_______________________________________________ > > > > >Swift-devel mailing list > > > > >Swift-devel at ci.uchicago.edu > > > > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > From itf at mcs.anl.gov Tue Apr 10 13:13:14 2007 From: itf at mcs.anl.gov (=?UTF-8?B?SWFuIEZvc3Rlcg==?=) Date: Tue, 10 Apr 2007 18:13:14 +0000 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <1176227815.12375.5.camel@blabla.mcs.anl.gov> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1195630042-1176227343-cardhu_blackberry.rim.net-1925209672-@bwe053-cell00.bisx.prod.on.blackberry> <1176227815.12375.5.camel@blabla.mcs.anl.gov> Message-ID: <2009955249-1176228969-cardhu_blackberry.rim.net-1332179839-@bwe005-cell00.bisx.prod.on.blackberry> Ok. I hope that can all come with a concise statement of how to do what is wanted here, as I can't follow these emails. Sounds like a good example for the doc. Sent via BlackBerry from T-Mobile -----Original Message----- From: Mihael Hategan Date: Tue, 10 Apr 2007 12:56:55 To:itf at mcs.anl.gov Cc:Veronika Nefedova , swift-devel-bounces at ci.uchicago.edu, Tibi Stef-Praun , swift-devel at ci.uchicago.edu Subject: Re: [Swift-devel] Interesting observation when running Swift On Tue, 2007-04-10 at 17:46 +0000, Ian Foster wrote: > I thought we could already do that. We can, I think, in the case of the first stage--we can take all files in a directory as a datset, say? But we can't do that for latter stages? Not much to do with when the stages happen. It's more about knowing what to stage out. Right now mappers can tell what files are relevant in a given collection (directory) that they manage, and the details are left to the implementations of the mappers. What's needed is to feed each mapper for return values from an atomic proc with the list of files generated by an application, let them select the relevant files, then do the stage-out and populate the Swift data structures based on that. > > > Sent via BlackBerry from T-Mobile > > -----Original Message----- > From: "Veronika V. Nefedova" > Date: Tue, 10 Apr 2007 12:12:12 > To:"Tiberiu Stef-Praun" , "Mihael Hategan" > Cc:swift-devel at ci.uchicago.edu > Subject: Re: [Swift-devel] Interesting observation when running Swift > > I think that something like that would be useful: > > outputsStage1[]=stage1() > outputsStage2[]=stage2(outputsStage1[]) > > if you didn't have to specify the number or specific filenames for the > outputs. Basically it would be good for the Workflow engine to understand > this: "get all the produced files from stage 1 and use them as an input for > Stage 2" > > (; > > Nika > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > >Interesting. > > > >Does anyone else think that monitoring the filesystem could be a useful idea ? > > > >For instance it could help with file-driven dependencies, in scenarios > >where we want to have continuous workflows, or compose independent > >wokflows. The filesystem would act as the publish-subscribe mechanism > >for some workflow cases. > > > >Tibi > > > >On 4/10/07, Mihael Hategan wrote: > >>Swift doesn't monitor the file system. > >>Data driven doesn't mean that it does magic in the background. It means > >>that you have to express data dependencies in the code. > >> > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > >> > I have a workflow along these lines: > >> > > >> > // this one generates outputsStage1[] > >> > stage1() > >> > // this one merges the stage1 outputs > >> > stage2(outputsStage1[]) > >> > > >> > note that it is not outputsStage1=stage1() > >> > > >> > Since the outputsStage1 files were not generated yet, I expected that > >> > Karajan waited for them to be created before running stage2, but that > >> > was not the case: stage2 was executed when the workflow started (and > >> > it failed) and caused the workflow to fail. > >> > > >> > I know how to fix the workflow, that is not the issue. The issue is > >> > that I expected the workflow to be data-driven, but it seems to be > >> > code driven. Explanation: it attempted to execute a section even if > >> > its input files were not available. > >> > > >> > Correct me if I am wrong. > >> > Tibi > >> > > >> > > > > > >-- > >Tiberiu (Tibi) Stef-Praun, PhD > >Research Staff, Computation Institute > >5640 S. Ellis Ave, #405 > >University of Chicago > >http://www-unix.mcs.anl.gov/~tiberius/ > >_______________________________________________ > >Swift-devel mailing list > >Swift-devel at ci.uchicago.edu > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Tue Apr 10 13:19:49 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Apr 2007 13:19:49 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1176225786.10451.25.camel@blabla.mcs.anl.gov> <1176228211.12875.0.camel@blabla.mcs.anl.gov> Message-ID: <1176229189.13194.9.camel@blabla.mcs.anl.gov> On Tue, 2007-04-10 at 13:08 -0500, Yong Zhao wrote: > something Mike has been calling collectors for a long time: which does > pattern matching (or other operations) on the remote execution directory > and collect the outputs etc. Yes, and it makes sense. Note that the difference between the existing() function of the mappers and the above would be the ability to work on a supplied file tree: existing(stuff), assuming that we don't want to clutter "collectors" with knowledge of the implementation of the remote file system access method. > > On Tue, 10 Apr 2007, Mihael Hategan wrote: > > > Remote mappers? > > > > On Tue, 2007-04-10 at 13:01 -0500, Yong Zhao wrote: > > > aren't these already addressed by remote mappers? > > > > > > Yong. > > > > > > On Tue, 10 Apr 2007, Mihael Hategan wrote: > > > > > > > On Tue, 2007-04-10 at 12:12 -0500, Veronika V. Nefedova wrote: > > > > > I think that something like that would be useful: > > > > > > > > > > outputsStage1[]=stage1() > > > > > outputsStage2[]=stage2(outputsStage1[]) > > > > > > > > > > if you didn't have to specify the number or specific filenames for the > > > > > outputs. Basically it would be good for the Workflow engine to understand > > > > > this: "get all the produced files from stage 1 and use them as an input for > > > > > Stage 2" > > > > > > > > Some applications are known to produce extra temporary files. Mappers > > > > are supposed to be able to extract arrays from a cluttered file system > > > > (assuming that there are no ambiguities in naming patterns). So yes, it > > > > would be useful, and I think one of the planned features, and I see no > > > > obvious problems besides the possible naming conflicts (which would > > > > apply to a local file system anyway, so not a new problem). > > > > > > > > > > > > > > (; > > > > > > > > > > Nika > > > > > > > > > > > > > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > > > > > >Interesting. > > > > > > > > > > > >Does anyone else think that monitoring the filesystem could be a useful idea ? > > > > > > > > > > > >For instance it could help with file-driven dependencies, in scenarios > > > > > >where we want to have continuous workflows, or compose independent > > > > > >wokflows. The filesystem would act as the publish-subscribe mechanism > > > > > >for some workflow cases. > > > > > > > > > > > >Tibi > > > > > > > > > > > >On 4/10/07, Mihael Hategan wrote: > > > > > >>Swift doesn't monitor the file system. > > > > > >>Data driven doesn't mean that it does magic in the background. It means > > > > > >>that you have to express data dependencies in the code. > > > > > >> > > > > > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > > > > > >> > I have a workflow along these lines: > > > > > >> > > > > > > >> > // this one generates outputsStage1[] > > > > > >> > stage1() > > > > > >> > // this one merges the stage1 outputs > > > > > >> > stage2(outputsStage1[]) > > > > > >> > > > > > > >> > note that it is not outputsStage1=stage1() > > > > > >> > > > > > > >> > Since the outputsStage1 files were not generated yet, I expected that > > > > > >> > Karajan waited for them to be created before running stage2, but that > > > > > >> > was not the case: stage2 was executed when the workflow started (and > > > > > >> > it failed) and caused the workflow to fail. > > > > > >> > > > > > > >> > I know how to fix the workflow, that is not the issue. The issue is > > > > > >> > that I expected the workflow to be data-driven, but it seems to be > > > > > >> > code driven. Explanation: it attempted to execute a section even if > > > > > >> > its input files were not available. > > > > > >> > > > > > > >> > Correct me if I am wrong. > > > > > >> > Tibi > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >-- > > > > > >Tiberiu (Tibi) Stef-Praun, PhD > > > > > >Research Staff, Computation Institute > > > > > >5640 S. Ellis Ave, #405 > > > > > >University of Chicago > > > > > >http://www-unix.mcs.anl.gov/~tiberius/ > > > > > >_______________________________________________ > > > > > >Swift-devel mailing list > > > > > >Swift-devel at ci.uchicago.edu > > > > > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > From hategan at mcs.anl.gov Tue Apr 10 13:24:41 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Apr 2007 13:24:41 -0500 Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <2009955249-1176228969-cardhu_blackberry.rim.net-1332179839-@bwe005-cell00.bisx.prod.on.blackberry> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1195630042-1176227343-cardhu_blackberry.rim.net-1925209672-@bwe053-cell00.bisx.prod.on.blackberry> <1176227815.12375.5.camel@blabla.mcs.anl.gov> <2009955249-1176228969-cardhu_blackberry.rim.net-1332179839-@bwe005-cell00.bisx.prod.on.blackberry> Message-ID: <1176229481.13194.14.camel@blabla.mcs.anl.gov> On Tue, 2007-04-10 at 18:13 +0000, Ian Foster wrote: > Ok. I hope that can all come with a concise statement of how to do what is wanted here, as I can't follow these emails. These things seem to have a tendency to become very fuzzy as you get into the details, which is probably why there's no actual implementation yet. > > Sounds like a good example for the doc. > > > Sent via BlackBerry from T-Mobile > > -----Original Message----- > From: Mihael Hategan > Date: Tue, 10 Apr 2007 12:56:55 > To:itf at mcs.anl.gov > Cc:Veronika Nefedova , swift-devel-bounces at ci.uchicago.edu, Tibi Stef-Praun , swift-devel at ci.uchicago.edu > Subject: Re: [Swift-devel] Interesting observation when running Swift > > On Tue, 2007-04-10 at 17:46 +0000, Ian Foster wrote: > > I thought we could already do that. We can, I think, in the case of the first stage--we can take all files in a directory as a datset, say? But we can't do that for latter stages? > > Not much to do with when the stages happen. It's more about knowing what > to stage out. > > Right now mappers can tell what files are relevant in a given collection > (directory) that they manage, and the details are left to the > implementations of the mappers. What's needed is to feed each mapper for > return values from an atomic proc with the list of files generated by an > application, let them select the relevant files, then do the stage-out > and populate the Swift data structures based on that. > > > > > > > Sent via BlackBerry from T-Mobile > > > > -----Original Message----- > > From: "Veronika V. Nefedova" > > Date: Tue, 10 Apr 2007 12:12:12 > > To:"Tiberiu Stef-Praun" , "Mihael Hategan" > > Cc:swift-devel at ci.uchicago.edu > > Subject: Re: [Swift-devel] Interesting observation when running Swift > > > > I think that something like that would be useful: > > > > outputsStage1[]=stage1() > > outputsStage2[]=stage2(outputsStage1[]) > > > > if you didn't have to specify the number or specific filenames for the > > outputs. Basically it would be good for the Workflow engine to understand > > this: "get all the produced files from stage 1 and use them as an input for > > Stage 2" > > > > (; > > > > Nika > > > > > > At 11:57 AM 4/10/2007, Tiberiu Stef-Praun wrote: > > >Interesting. > > > > > >Does anyone else think that monitoring the filesystem could be a useful idea ? > > > > > >For instance it could help with file-driven dependencies, in scenarios > > >where we want to have continuous workflows, or compose independent > > >wokflows. The filesystem would act as the publish-subscribe mechanism > > >for some workflow cases. > > > > > >Tibi > > > > > >On 4/10/07, Mihael Hategan wrote: > > >>Swift doesn't monitor the file system. > > >>Data driven doesn't mean that it does magic in the background. It means > > >>that you have to express data dependencies in the code. > > >> > > >>On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote: > > >> > I have a workflow along these lines: > > >> > > > >> > // this one generates outputsStage1[] > > >> > stage1() > > >> > // this one merges the stage1 outputs > > >> > stage2(outputsStage1[]) > > >> > > > >> > note that it is not outputsStage1=stage1() > > >> > > > >> > Since the outputsStage1 files were not generated yet, I expected that > > >> > Karajan waited for them to be created before running stage2, but that > > >> > was not the case: stage2 was executed when the workflow started (and > > >> > it failed) and caused the workflow to fail. > > >> > > > >> > I know how to fix the workflow, that is not the issue. The issue is > > >> > that I expected the workflow to be data-driven, but it seems to be > > >> > code driven. Explanation: it attempted to execute a section even if > > >> > its input files were not available. > > >> > > > >> > Correct me if I am wrong. > > >> > Tibi > > >> > > > >> > > > > > > > > >-- > > >Tiberiu (Tibi) Stef-Praun, PhD > > >Research Staff, Computation Institute > > >5640 S. Ellis Ave, #405 > > >University of Chicago > > >http://www-unix.mcs.anl.gov/~tiberius/ > > >_______________________________________________ > > >Swift-devel mailing list > > >Swift-devel at ci.uchicago.edu > > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > From hategan at mcs.anl.gov Wed Apr 11 14:45:33 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 11 Apr 2007 14:45:33 -0500 Subject: [Swift-devel] Re: [Swift-user] MolDyn workflow failing on GPFS at NCSA In-Reply-To: <461D31A2.4080403@mcs.anl.gov> References: <6.0.0.22.2.20070411131554.03a8be80@mail.mcs.anl.gov> <461D31A2.4080403@mcs.anl.gov> Message-ID: <1176320733.14524.13.camel@blabla.mcs.anl.gov> [moving to swift-devel] On Wed, 2007-04-11 at 14:06 -0500, Mike Wilde wrote: > Hi all, > > I'm moving this discussion to swift-devel. This concerns a problem that is > preventing Nika's MolDyn workflow from running on the NCSA TG IA64 cluster > under GPFS. > [...] > So lets pust forward and solve this one. Here's my questions: > > 0) What do we know about the problem from tests to date? Is the problem (the > timeout?) due to not getting a job-completion notification (as it appears form > the log below) or a data transfer timeout (as I got the possibly mistaken > impression from my discussion yesterday with Nika)? > > 1) As its only been run at NCSA, I suggested yesterday that we take more > advantage of the uc-teragrid node - less loaded, and we have very capable > eager staff at Argonne to help us debug. It would also help us better assess > whether we are looking at a GPFS problem, a local NCSA problem, a Globus > problem, or a Swift problem. I dont see any conclusive evidence yet that this > problem is specifically in GPFS or that GPFS is behaving incorrectly. Me neither. And I find it difficult to see what the file system has to do with the outcome that we see. From the runs Nika did so far, it seems that there is a correlation. I am however concerned about correlations inferred from a low number of experiments. > > 2) It might be worth a try to isolate the problem with simpler Swift code, but > that will take time and we have no guarantee up-front *how* much time. Could > be a long hunt. Do we have a clue as to what pattern to start with in simpler > Swift code? Could we dial-down the job duration of this run - way down - so > that we can make the problem happen quicker, with less waiting? Its more > important that we can make it happen quickly and repeatedly than that we do it > with simpler code. The latter would be nice but seems to be not a necessity > (yet). Sorry. I wrote that with the assumption that speed and reproducibility are closely related to simpler code. > > 3) Do we have enough logging turned on in the current runs to isolate the > problem a bit further? Do we need to insert additional logging Don't know. From my analysis of the logs, I couldn't tell that there is a abnormal problem. Other workflows finish successfully similar log patterns. What I also instructed Nika to do is to always run this with remote debugging enabled. That way I can hook up to the running process and get more details that would otherwise be hard to see. > > 4) Does this problem bear any relation to observed pre-WS-GRAM problems in > getting back job completion notification on NFS sites? These are, eg, > problems that Jen got deeply into with VDS/Pegasus/DAGman. He made mods to > the PBS and COndor job managers to support reduce a suspected NFS-aggravated > race condition. Could there be a similar issue in the NCSA Globus PBS > jobmanager? (Note that Jens's issues were with NFS, but similar phenomena > could plague any shared FS that globus depends on for completion > synchronization). This problem was, to my knowldege, never fully resolved. It's all cool, but the job manager stores its files on the same filesystem. GPFS vs. home refers to the swift temporary directories, which should be a separate concern (and this is what makes it odd, since the expected behavior in case of problems would be missing file errors not notification issues). > > 5) Could we re-run this workflow using WS-GRAM? I think that would only complicate the issue since it's harder to debug WS-GRAM than pre-WS GRAM. > > 6) I see notes in the thread below that say the problem happens only on GPFS, > and other notes that it happens also on the /home filesystem. Whats the > current knowledge on this? Have you determined conclusively that this is a > GPFS-only problem? No. See above. Mihael > > Thanks, > > Mik > > > Veronika V. Nefedova wrote, On 4/11/2007 1:16 PM: > > Mike, This is the thread that Mihael was involved with (with help at teragrid) > > > > Nika > > > > > >> Date: Mon, 9 Apr 2007 22:15:36 -0500 > >> Subject: Re: workflow not working on GPFS > >> To: "Veronika V. Nefedova" > >> From: help at teragrid.org > >> Cc: > >> X-Mailer: Perl5 Mail::Internet v1.74 > >> Sender: Nobody > >> X-Null-Tag: b6f18422934f93a28d94a359b5c1b409 > >> X-NCSA-MailScanner-Information: Please contact help at ncsa.uiuc.edu for > >> more information, amantadine.ncsa.uiuc.edu > >> X-NCSA-MailScanner: Found to be clean > >> X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at > >> mailgw.mcs.anl.gov > >> > >> FROM: McWilliams, David G > >> (Concerning ticket No. 138013) > >> > >> Nika, > >> > >> I asked the system administrator to create a directory for you. I will > >> let you > >> know when its been created. > >> > >> Dave McWilliams (217) 244-1144 consult at ncsa.uiuc.edu > >> NCSA Consulting Services http://www.ncsa.uiuc.edu/UserInfo/Consulting/ > >> -------------------------------------------------------------------------- > >> > >> > >> Veronika V. Nefedova writes: > >> >I have some preliminary number. Its 121.7MB per molecule. We would > >> like to > >> >do 250 molecules at once (which is our goal), but for now it would be > >> very > >> >good to be able to do 50 molecules at a time. > >> >What do I need to do in order to have an access to such disk space ? > >> > > >> >Thanks, > >> > > >> >Nika > >> > > >> >At 03:25 PM 4/9/2007, Veronika V. Nefedova wrote: > >> >>yep, my main purpose for using gpfs was the disk space. And its only a > >> >>temporary disk space (for the duration of the job run time), not for > >> >>storage. Let me try to estimate how much space I need (as I > >> mentioned - my > >> >>directories got removed once the job finishes successfully). > >> >> > >> >>Thanks! > >> >> > >> >>Nika > >> >> > >> >>At 03:15 PM 4/9/2007, help at teragrid.org wrote: > >> >>>FROM: McWilliams, David G > >> >>>(Concerning ticket No. 138013) > >> >>> > >> >>>Nika, > >> >>> > >> >>>Is it critical that you run out of /gpfs or do you just need more > >> space > >> >>>than is > >> >>>available in your home directory? Another alternative would be to > >> request > >> >>>space > >> >>>in the /usr/projects directory which is the same type filesystem as > >> the home > >> >>>directory filesystem. If that would work, how much disk space do > >> you need? > >> >>> > >> >>>Dave McWilliams (217) 244-1144 > >> consult at ncsa.uiuc.edu > >> >>>NCSA Consulting Services > >> http://www.ncsa.uiuc.edu/UserInfo/Consulting/ > >> >>>-------------------------------------------------------------------------- > >> > >> >>> > >> >>>Veronika V. Nefedova writes: > >> >>> >Hi, Dave: > >> >>> > > >> >>> >OK, I do not have single successful run on /gpfs system of my > >> workflow. I > >> >>> >ran the same exact job on /home and it worked just fine. > >> >>> >Today the 3-rd job of the workflow failed because of the timeout > >> (thats my > >> >>> >guess, it never timeouts on /home). Last time I tried on gpfs > >> that job > >> >>> >finished just fine and in fact that were the next jobs in the > >> workflow > >> >>> that > >> >>> >failed. Something is wrong with gpfs it seems. > >> >>> > > >> >>> >Thanks, > >> >>> > > >> >>> >Nika > >> >>> > > >> >>> > > >> >>> >At 05:06 PM 4/4/2007, Mihael Hategan wrote: > >> >>> >>Nika reports that some jobs have finished correctly. This seems to > >> >>> >>indicate that things are OK on the Globus side, and that it's a > >> problem > >> >>> >>with the specification. > >> >>> >> > >> >>> >>Mihael > >> >>> >> > >> >>> >>On Wed, 2007-04-04 at 17:03 -0500, Veronika V. Nefedova wrote: > >> >>> >> > Hi, Dave: > >> >>> >> > > >> >>> >> > the workflow engine is waiting for the completion > >> notification from > >> >>> Globus. > >> >>> >> > Now all my 68 jobs have finished, but no notification was sent > >> >>> back. So > >> >>> >> the > >> >>> >> > workflow doesn't submit any new jobs. I am Cc Mihael who > >> might be > >> >>> able to > >> >>> >> > suggest what is going on wrong here. > >> >>> >> > > >> >>> >> > Could the problem be that telnet problem that happened earlier > >> >>> today ? My > >> >>> >> > jobs were already submitted, but not finished yet. (and I got > >> >>> kicked off > >> >>> >> > from the login node) > >> >>> >> > > >> >>> >> > Thanks for looking into this. > >> >>> >> > > >> >>> >> > Nika > >> >>> >> > > >> >>> >> > At 03:46 PM 4/4/2007, help at teragrid.org wrote: > >> >>> >> > >FROM: McWilliams, David G > >> >>> >> > >(Concerning ticket No. 138013) > >> >>> >> > > > >> >>> >> > >Nika, > >> >>> >> > > > >> >>> >> > >Thanks for the additional information. It is good to know > >> that the > >> >>> >> problem is > >> >>> >> > >not specific to jobs that use GPFS. How does the workflow > >> engine know > >> >>> >> that the > >> >>> >> > >job is done? Does that workflow engine wait for the globusrun > >> >>> command to > >> >>> >> > >return > >> >>> >> > >or is it looking in a file for the job status? > >> >>> >> > > > >> >>> >> > >Dave McWilliams (217) > >> >>> 244-1144 consult at ncsa.uiuc.edu > >> >>> >> > >NCSA Consulting > >> >>> Services http://www.ncsa.uiuc.edu/UserInfo/Consulting/ > >> >>> >> > > >> >---------------------------------------------------------------- --- > >> >>> ------- > >> >>> >> > > > >> >>> >> > >Veronika V. Nefedova writes: > >> >>> >> > > >Actually, this current job is failing despite the workdir > >> set to > >> >>> >> /home. I > >> >>> >> > > >had 68 jobs (75 mins each) in the queue. I see that I have > >> now 19 > >> >>> >> running > >> >>> >> > > >and 27 queued jobs, which means that 68-19-27=22 jobs have > >> already > >> >>> >> finished > >> >>> >> > > >(the output from those jobs confirms it) but none of these > >> >>> finished > >> >>> >> jobs > >> >>> >> > > >are marked as 'done' on my side. I.e. gram didn't send the > >> >>> >> completion info > >> >>> >> > > >back to the submit side. So the problem is not unique to > >> gpfs... > >> >>> >> > > > > >> >>> >> > > >Nika > >> >>> >> > > > > >> >>> >> > > >At 02:00 PM 4/4/2007, Veronika V. Nefedova wrote: > >> >>> >> > > >>Hi, Dave: > >> >>> >> > > >> > >> >>> >> > > >>The jobs are submitted in the form of RSL via API calls. > >> So I do > >> >>> >> not have > >> >>> >> > > >>scripts per se. This what I have in my log file: > >> >>> >> > > >> > >> >>> >> > > >>2007-04-04 13:15:51,653 DEBUG JobSubmissionTaskHandler > >> RSL: &( > >> >>> >> directory = > >> >>> >> > > >>"/home/ac/nefed > >> >>> >> > > >>ova/SWIFT/MolDyn-ty2oh8r2ki171" )( arguments = > >> "shared/wrapper.sh" > >> >>> >> > > >>"chrm_long-720q4k9i" "s > >> >>> >> > > >>olv_repu_0.8_0.9_b0_m001.out" "stderr.txt" "solv.inp" "" > >> "solv.inp > >> >>> >> > > >>parm03_gaffnb_all.prm p > >> >>> >> > > >>arm03_gaff_all.rtf m001_am1.rtf m001_am1.prm solv_m001.psf > >> >>> >> > > >>solv_m001_eq.crd solv_repu_0.8_ > >> >>> >> > > >>0.9_b0.prt" "solv_repu_0.8_0.9_b0_m001.wham > >> >>> >> solv_repu_0.8_0.9_b0_m001.crd > >> >>> >> > > >>solv_repu_0.8_0. > >> >>> >> > > >>9_b0_m001.out solv_repu_0.8_0.9_b0_m001_done" "" > >> >>> >> > > >>"/home/ac/yqdeng/c34a2/exec/altix/charmm" > >> >>> >> > > >> "pstep:40000" "prtfile:solv_repu_0.8_0.9_b0" > >> "system:solv_m001" > >> >>> >> > > >> "stitle:m001" "rtffile:pa > >> >>> >> > > >>rm03_gaff_all.rtf" "paramfile:parm03_gaffnb_all.prm" > >> >>> "gaff:m001_am1" > >> >>> >> > > >>"vac:" "restart:NONE" > >> >>> >> > > >> "faster:off" "rwater:15" "chem:chem" "minstep:0" > >> "rforce:0" > >> >>> >> "ligcrd:lyz" > >> >>> >> > > >> "stage:repu" "ur > >> >>> >> > > >>andseed:2640378" "dirname:solv_repu_0.8_0.9_b0_m001" > >> "rcut1:0.8" > >> >>> >> > > >>"rcut2:0.9" )( executable > >> >>> >> > > >> = "/bin/sh" )( maxwalltime = "75" )( environment = ( > >> "PATH" > >> >>> >> > > >> "/home/ac/nefedova/bin/tools/ > >> >>> >> > > >>bin/:/usr/bin:/bin" ) ) > >> >>> >> > > >> > >> >>> >> > > >>the working directory is specified at the very beginning ( > >> >>> directory = > >> >>> >> > > >>"/home/ac/nefedova/SWIFT/MolDyn-ty2oh8r2ki171" ). When I > >> try to > >> >>> specify > >> >>> >> > > >>the /gpfs_scratch1/nefedova/bla -- the 'finished' status > >> >>> doesn't return > >> >>> >> > > >>back to my workflow engine. Right now I have the working > >> >>> version of the > >> >>> >> > > >>workflow running (in /home). > >> >>> >> > > >> > >> >>> >> > > >>Thank you very much for looking into this! > >> >>> >> > > >> > >> >>> >> > > >>Nika > >> >>> >> > > >> > >> >>> >> > > >>At 07:51 PM 4/3/2007, help at teragrid.org wrote: > >> >>> >> > > >>>FROM: McWilliams, David G > >> >>> >> > > >>>(Concerning ticket No. 138013) > >> >>> >> > > >>> > >> >>> >> > > >>>Nika, > >> >>> >> > > >>> > >> >>> >> > > >>>I would suggest that we start by comparing the PBS batch > >> scripts > >> >>> >> that are > >> >>> >> > > >>>created for the jobs with both working directories. Does > >> the > >> >>> workflow > >> >>> >> > > engine > >> >>> >> > > >>>report the PBS job IDs? If so, please send the job IDs > >> for both > >> >>> >> job types? > >> >>> >> > > >>> > >> >>> >> > > >>>Dave McWilliams (217) > >> >>> >> 244-1144 consult at ncsa.uiuc.edu > >> >>> >> > > >>>NCSA Consulting > >> >>> >> Services http://www.ncsa.uiuc.edu/UserInfo/Consulting/ > >> >>> >> > > > >> >>>------------------------------------------------------------ --- > >> >>> ---- > >> >>> >> ------- > >> >>> >> > > >>> > >> >>> >> > > >>>Veronika V. Nefedova writes: > >> >>> >> > > >>> >Hi, > >> >>> >> > > >>> > > >> >>> >> > > >>> >I am using Swift workflow engine > >> >>> >> (http://www.ci.uchicago.edu/swift/) to > >> >>> >> > > >>> >submit my jobs to Teragrid/NCSA from my machine at ANL. > >> >>> >> > > >>> >I have some problems with having my workflow ran on > >> /gpfs on > >> >>> >> Teragrid. > >> >>> >> > > >>> When > >> >>> >> > > >>> >my working directory is set to /home, the workflow works > >> >>> just fine. > >> >>> >> > > When I > >> >>> >> > > >>> >want to run a large run I set my workdir to /gpfs > >> (which has > >> >>> a lot > >> >>> >> > > of disk > >> >>> >> > > >>> >space) and then my workflow stops working. The very > >> >>> beginning of the > >> >>> >> > > >>> >workflow (that has some small jobs) is working ok, but > >> after the > >> >>> >> > > long jobs > >> >>> >> > > >>> >(66 min each) are finished the system never returns > >> the 'done' > >> >>> >> status > >> >>> >> > > >>> back > >> >>> >> > > >>> >to Swift (@ANL) and thus no new jobs are submitted and > >> the > >> >>> workflow > >> >>> >> > > stops. > >> >>> >> > > >>> >Again, exactly the same workflow works fine when > >> workdir is > >> >>> set to > >> >>> >> > > /home. > >> >>> >> > > >>> > > >> >>> >> > > >>> >Thanks! > >> >>> >> > > >>> > > >> >>> >> > > >>> >Nika > >> >>> >> > > >> >>> >> > > > > > > > > From benc at hawaga.org.uk Wed Apr 11 23:58:57 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 12 Apr 2007 11:58:57 +0700 (ICT) Subject: [Swift-devel] print misbehavin' Message-ID: I think the below program should print the string "hello". But it doesn't - instead it outputs "null". This is with r613. Bleugh. $ cat print.swift print("hello"); $ swift print.swift Swift V 0.0405 RunID: 9oiyuyeb6f6r0 null -- From benc at hawaga.org.uk Wed Apr 11 23:58:03 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 12 Apr 2007 11:58:03 +0700 (ICT) Subject: [Swift-devel] print. Message-ID: I think the below program should print the string "hello". But it doesn't - instead it outputs "null". This is with r613. Bleugh. $ cat print.swift print("hello"); $ swift print.swift Swift V 0.0405 RunID: 9oiyuyeb6f6r0 null -- From hategan at mcs.anl.gov Thu Apr 12 08:54:09 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 12 Apr 2007 08:54:09 -0500 Subject: [Swift-devel] print misbehavin' In-Reply-To: References: Message-ID: <1176386049.7871.1.camel@blabla.mcs.anl.gov> It translates fine though. The issue is Swift data and their toString(). On Thu, 2007-04-12 at 11:58 +0700, Ben Clifford wrote: > I think the below program should print the string "hello". But it doesn't > - instead it outputs "null". This is with r613. Bleugh. > > > $ cat print.swift > > > print("hello"); > > > > $ swift print.swift > Swift V 0.0405 > RunID: 9oiyuyeb6f6r0 > null > From benc at hawaga.org.uk Thu Apr 12 23:00:23 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 13 Apr 2007 11:00:23 +0700 (ICT) Subject: [Swift-devel] print misbehavin' In-Reply-To: <1176386049.7871.1.camel@blabla.mcs.anl.gov> References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> Message-ID: On Thu, 12 Apr 2007, Mihael Hategan wrote: > It translates fine though. > The issue is Swift data and their toString(). So, turns out that: print(3) will not work, but print(1+2) will work. What happens is that '1' is a constant, which goes through a different code path in Karajan.java to '1+2' which is an expression, generating these two kml fragments: vs 1 2 So it seems that at least in the simple case, the vdl:new construct is unnecessary for basic expressions. I tried removing the special handling for constants in Karajan.java so that '3' goes through the same code path as '1+2' and so print(3) turns into this code fragment instead: 3 which successfully prints out 3. Hurrah! I haven't really thought about if this is going to break other more complicated stuff yet, so I'm not going to commit this patch yet. But I am less frustrated than yesterday ;-) -- From hategan at mcs.anl.gov Thu Apr 12 23:12:21 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 12 Apr 2007 23:12:21 -0500 Subject: [Swift-devel] print misbehavin' In-Reply-To: References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> Message-ID: <1176437541.30675.6.camel@blabla.mcs.anl.gov> Actually that will break user defined procs, since they expect wrapped values. And the translation of + to sum() is also weird since it will break if you have any wrapped values one either side. I mean there are two choices here. There are swift values and non-swift values. Using them with a plain karajan function requires them to be unwrapped (vdl:getfieldvalue) and the result wrapped back with vdl:new. Alternatively the function can be overriden to work on swift values. And now that I see it, maybe some simple reduction would be useful. parallel(3) is a useless alias for 3. Mihael On Fri, 2007-04-13 at 11:00 +0700, Ben Clifford wrote: > On Thu, 12 Apr 2007, Mihael Hategan wrote: > > > It translates fine though. > > The issue is Swift data and their toString(). > > So, turns out that: > > print(3) will not work, but > > print(1+2) will work. > > What happens is that '1' is a constant, which goes through a different > code path in Karajan.java to '1+2' which is an expression, generating > these two kml fragments: > > > > > > > > > > vs > > > > > 1 > 2 > > > > > > So it seems that at least in the simple case, the vdl:new construct is > unnecessary for basic expressions. I tried removing the special handling > for constants in Karajan.java so that '3' goes through the same code path > as '1+2' and so print(3) turns into this code fragment instead: > > > > 3 > > > > > which successfully prints out 3. Hurrah! > > I haven't really thought about if this is going to break other more > complicated stuff yet, so I'm not going to commit this patch yet. But I am > less frustrated than yesterday ;-) > From benc at hawaga.org.uk Fri Apr 13 00:25:43 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 13 Apr 2007 12:25:43 +0700 (ICT) Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <1176229481.13194.14.camel@blabla.mcs.anl.gov> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1195630042-1176227343-cardhu_blackberry.rim.net-1925209672-@bwe053-cell00.bisx.prod.on.blackberry> <1176227815.12375.5.camel@blabla.mcs.anl.gov> <2009955249-1176228969-cardhu_blackberry.rim.net-1332179839-@bwe005-cell00.bisx.prod.on.blackberry> <1176229481.13194.14.camel@blabla.mcs.anl.gov> Message-ID: On Tue, 10 Apr 2007, Mihael Hategan wrote: > These things seem to have a tendency to become very fuzzy as you get > into the details, which is probably why there's no actual implementation > yet. I've tried to experiment with implementation. I find myself frustrated by various other problems on the way (such as cracked out expression evaluation) that constantly sidetracks me. Hopefully we'll get there eventually. -- From benc at hawaga.org.uk Fri Apr 13 00:25:49 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 13 Apr 2007 12:25:49 +0700 (ICT) Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: <6.0.0.22.2.20070410130311.03a19350@mail.mcs.anl.gov> References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1176225786.10451.25.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410130311.03a19350@mail.mcs.anl.gov> Message-ID: On Tue, 10 Apr 2007, Veronika V. Nefedova wrote: > do we have the remote mappers available now ? I didn't see any > announcement... no. no where near implementation. -- From benc at hawaga.org.uk Fri Apr 13 00:26:43 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 13 Apr 2007 12:26:43 +0700 (ICT) Subject: [Swift-devel] Interesting observation when running Swift In-Reply-To: References: <1176223855.10019.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070410120824.05059c10@mail.mcs.anl.gov> <1176225786.10451.25.camel@blabla.mcs.anl.gov> Message-ID: On Tue, 10 Apr 2007, Yong Zhao wrote: > aren't these already addressed by remote mappers? depends what you mean by 'already'. The concept of remote mappers pretty much addresses this, i think; but there is nothing implementation-wise, and I don't believe our present code base is sufficiently stable/coherent at the moment to really support much interesting experimentation. -- From benc at hawaga.org.uk Wed Apr 18 04:01:20 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 18 Apr 2007 09:01:20 +0000 (GMT) Subject: [Swift-devel] print misbehavin' In-Reply-To: <1176437541.30675.6.camel@blabla.mcs.anl.gov> References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> <1176437541.30675.6.camel@blabla.mcs.anl.gov> Message-ID: On Thu, 12 Apr 2007, Mihael Hategan wrote: > Actually that will break user defined procs, since they expect wrapped > values. > And the translation of + to sum() is also weird since it will break if > you have any wrapped values one either side. ok. I didn't commit that. In r620 I put in some different 'toString()' handling for the cases where AbstractDataNode.toString() was returning null -- it now provides some descriptive information instead: print(3) gives (instead of the previous "null"): unnamed SwiftScript value with value 3 and print(1+2) still gives (as previously): 3 This isn't the behaviour that I'd like in the long term, but I haven't really thought about what the right way to proceed there is - either your suggestions below, or removing the presence of non-wrapped types entirely. > I mean there are two choices here. There are swift values and non-swift > values. Using them with a plain karajan function requires them to be > unwrapped (vdl:getfieldvalue) and the result wrapped back with vdl:new. > Alternatively the function can be overriden to work on swift values. -- From benc at hawaga.org.uk Wed Apr 18 07:14:25 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 18 Apr 2007 12:14:25 +0000 (GMT) Subject: [Swift-devel] print misbehavin' In-Reply-To: <1176437541.30675.6.camel@blabla.mcs.anl.gov> References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> <1176437541.30675.6.camel@blabla.mcs.anl.gov> Message-ID: On Thu, 12 Apr 2007, Mihael Hategan wrote: > Actually that will break user defined procs, since they expect wrapped > values. user defined procs are broken either way - at present, they can't take expressions are parameters: q=p(3) correctly outputs a 3 in something like this: type file {}; (file i) p(int n) { app { echo n stdout=@filename(i); } } file q <"volume.blah">; q=p(3); but saying q=p(1+2) breaks with this: Incompatible argument type for var: expected handle; got java.lang.Double. Offending argument: 3.0 for pretty much the same reason - its expecting a wrapped value but expression evaluation is providing an unwrapped value. bleugh. -- From yongzh at cs.uchicago.edu Wed Apr 18 10:23:25 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Wed, 18 Apr 2007 10:23:25 -0500 (CDT) Subject: [Swift-devel] print misbehavin' In-Reply-To: References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> <1176437541.30675.6.camel@blabla.mcs.anl.gov> Message-ID: Yeah, I wanted to wrap the karajan functions such as sum, product etc and return handles but then realized that karajan did not keep the type information. We need to find a way to make this right. Yong. On Wed, 18 Apr 2007, Ben Clifford wrote: > > > On Thu, 12 Apr 2007, Mihael Hategan wrote: > > > Actually that will break user defined procs, since they expect wrapped > > values. > > user defined procs are broken either way - at present, they can't take > expressions are parameters: > > q=p(3) correctly outputs a 3 in something like this: > > type file {}; > > (file i) p(int n) { > app { > echo n stdout=@filename(i); > } > } > > file q <"volume.blah">; > > q=p(3); > > but saying q=p(1+2) breaks with this: > > Incompatible argument type for var: expected handle; got java.lang.Double. > Offending argument: 3.0 > > for pretty much the same reason - its expecting a wrapped value but > expression evaluation is providing an unwrapped value. > > bleugh. > > -- > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Wed Apr 18 10:57:31 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 18 Apr 2007 10:57:31 -0500 Subject: [Swift-devel] print misbehavin' In-Reply-To: References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> <1176437541.30675.6.camel@blabla.mcs.anl.gov> Message-ID: <1176911851.27522.3.camel@blabla.mcs.anl.gov> On Wed, 2007-04-18 at 10:23 -0500, Yong Zhao wrote: > Yeah, I wanted to wrap the karajan functions such as sum, product etc > and return handles but then realized that karajan did not keep the type > information. It does not have a type system. Of course it doesn't keep type info, nor would it have polymorphism. But I don't think that's an issue. We've already established that from the beginning and created a swift specific type system. I don't see why that cannot be used. I.e. instead of sum, you translate to vdl:sum which unwraps the values, does the addition and wraps the result back. This is pretty much what any dynamic type system would do. > We need to find a way to make this right. > > Yong. > > On Wed, 18 Apr 2007, Ben Clifford wrote: > > > > > > > On Thu, 12 Apr 2007, Mihael Hategan wrote: > > > > > Actually that will break user defined procs, since they expect wrapped > > > values. > > > > user defined procs are broken either way - at present, they can't take > > expressions are parameters: > > > > q=p(3) correctly outputs a 3 in something like this: > > > > type file {}; > > > > (file i) p(int n) { > > app { > > echo n stdout=@filename(i); > > } > > } > > > > file q <"volume.blah">; > > > > q=p(3); > > > > but saying q=p(1+2) breaks with this: > > > > Incompatible argument type for var: expected handle; got java.lang.Double. > > Offending argument: 3.0 > > > > for pretty much the same reason - its expecting a wrapped value but > > expression evaluation is providing an unwrapped value. > > > > bleugh. > > > > -- > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > From yongzh at cs.uchicago.edu Wed Apr 18 11:59:48 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Wed, 18 Apr 2007 11:59:48 -0500 (CDT) Subject: [Swift-devel] print misbehavin' In-Reply-To: <1176911851.27522.3.camel@blabla.mcs.anl.gov> References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> <1176437541.30675.6.camel@blabla.mcs.anl.gov> <1176911851.27522.3.camel@blabla.mcs.anl.gov> Message-ID: I was referring to the mechnism to put extra stuff along with an object. anyways, if we change all the translations of constants to swift objects, we might be able to do this. But it is a complicated issue once you have all the paths and expressions and constants mixed together. Yong. On Wed, 18 Apr 2007, Mihael Hategan wrote: > On Wed, 2007-04-18 at 10:23 -0500, Yong Zhao wrote: > > Yeah, I wanted to wrap the karajan functions such as sum, product etc > > and return handles but then realized that karajan did not keep the type > > information. > > It does not have a type system. Of course it doesn't keep type info, nor > would it have polymorphism. > > But I don't think that's an issue. We've already established that from > the beginning and created a swift specific type system. I don't see why > that cannot be used. > I.e. instead of sum, you translate to vdl:sum which unwraps the values, > does the addition and wraps the result back. This is pretty much what > any dynamic type system would do. > > > We need to find a way to make this right. > > > > Yong. > > > > On Wed, 18 Apr 2007, Ben Clifford wrote: > > > > > > > > > > > On Thu, 12 Apr 2007, Mihael Hategan wrote: > > > > > > > Actually that will break user defined procs, since they expect wrapped > > > > values. > > > > > > user defined procs are broken either way - at present, they can't take > > > expressions are parameters: > > > > > > q=p(3) correctly outputs a 3 in something like this: > > > > > > type file {}; > > > > > > (file i) p(int n) { > > > app { > > > echo n stdout=@filename(i); > > > } > > > } > > > > > > file q <"volume.blah">; > > > > > > q=p(3); > > > > > > but saying q=p(1+2) breaks with this: > > > > > > Incompatible argument type for var: expected handle; got java.lang.Double. > > > Offending argument: 3.0 > > > > > > for pretty much the same reason - its expecting a wrapped value but > > > expression evaluation is providing an unwrapped value. > > > > > > bleugh. > > > > > > -- > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > From hategan at mcs.anl.gov Wed Apr 18 12:09:27 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 18 Apr 2007 12:09:27 -0500 Subject: [Swift-devel] print misbehavin' In-Reply-To: References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> <1176437541.30675.6.camel@blabla.mcs.anl.gov> <1176911851.27522.3.camel@blabla.mcs.anl.gov> Message-ID: <1176916167.27875.0.camel@blabla.mcs.anl.gov> On Wed, 2007-04-18 at 11:59 -0500, Yong Zhao wrote: > I was referring to the mechnism to put extra stuff along with an object. > anyways, if we change all the translations of constants to swift objects, > we might be able to do this. But it is a complicated issue once you have > all the paths and expressions and constants mixed together. I'm not following. > > Yong. > > On Wed, 18 Apr 2007, Mihael Hategan wrote: > > > On Wed, 2007-04-18 at 10:23 -0500, Yong Zhao wrote: > > > Yeah, I wanted to wrap the karajan functions such as sum, product etc > > > and return handles but then realized that karajan did not keep the type > > > information. > > > > It does not have a type system. Of course it doesn't keep type info, nor > > would it have polymorphism. > > > > But I don't think that's an issue. We've already established that from > > the beginning and created a swift specific type system. I don't see why > > that cannot be used. > > I.e. instead of sum, you translate to vdl:sum which unwraps the values, > > does the addition and wraps the result back. This is pretty much what > > any dynamic type system would do. > > > > > We need to find a way to make this right. > > > > > > Yong. > > > > > > On Wed, 18 Apr 2007, Ben Clifford wrote: > > > > > > > > > > > > > > > On Thu, 12 Apr 2007, Mihael Hategan wrote: > > > > > > > > > Actually that will break user defined procs, since they expect wrapped > > > > > values. > > > > > > > > user defined procs are broken either way - at present, they can't take > > > > expressions are parameters: > > > > > > > > q=p(3) correctly outputs a 3 in something like this: > > > > > > > > type file {}; > > > > > > > > (file i) p(int n) { > > > > app { > > > > echo n stdout=@filename(i); > > > > } > > > > } > > > > > > > > file q <"volume.blah">; > > > > > > > > q=p(3); > > > > > > > > but saying q=p(1+2) breaks with this: > > > > > > > > Incompatible argument type for var: expected handle; got java.lang.Double. > > > > Offending argument: 3.0 > > > > > > > > for pretty much the same reason - its expecting a wrapped value but > > > > expression evaluation is providing an unwrapped value. > > > > > > > > bleugh. > > > > > > > > -- > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > From benc at hawaga.org.uk Wed Apr 18 12:22:54 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 18 Apr 2007 17:22:54 +0000 (GMT) Subject: [Swift-devel] print misbehavin' In-Reply-To: References: <1176386049.7871.1.camel@blabla.mcs.anl.gov> <1176437541.30675.6.camel@blabla.mcs.anl.gov> <1176911851.27522.3.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 18 Apr 2007, Yong Zhao wrote: > I was referring to the mechnism to put extra stuff along with an object. > anyways, if we change all the translations of constants to swift > objects, we might be able to do this. But it is a complicated issue once > you have all the paths and expressions and constants mixed together. I think the whole expression/function compilation/evaluation path could do with a fairly serious tidy-up - it has funny behaviour now and its hard to understand how it works. -- From benc at hawaga.org.uk Wed Apr 18 12:54:32 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 18 Apr 2007 17:54:32 +0000 (GMT) Subject: [Swift-devel] strcat Message-ID: I've made some changes, uncommitted, that let you write something like this: === type file {}; (file i) p() { app { echo "hi" stdout=@filename(i); } } file q <"volume.blah">; file r ; q=p(); r=p(); === This will create two files, once called "volume.blah" and one called "foovolume.blah". The point of this is to support the use cases from Tibi and Nika that motivated bug 24, one of the 0.2 target features. So now I guess I should see if this functionality is actually helpful for them in terms of expressing those workflows more concisely... -- From wilde at mcs.anl.gov Wed Apr 18 13:12:01 2007 From: wilde at mcs.anl.gov (Mike Wilde) Date: Wed, 18 Apr 2007 13:12:01 -0500 Subject: [Swift-devel] strcat In-Reply-To: References: Message-ID: <46265F71.7000100@mcs.anl.gov> This sounds good, Ben. I had asked Nika and Tibi, as part of their application write-ups and wiki logs, to include pointers to the swift code, and in the writeups to also show not only the way its coded today but also the way we would *like* to code it. Ive also asked that we do all our app test runs, to the extent possible, from below a uniform directory structure on the CI NFS server. So this would let you find their examples and copy them to try new features such as this out on. In this case, I dont know whats easiest, and who has time to get to it first. As this is not checked in, it seems like for now what we need Ben is for you to get access to a runnable example, and recode it using @strcat(). If this sounds reasonable, then Nika and Tibi, could you each set up one simple but working example for Ben to try, on terminable? Thanks, Mike Ben Clifford wrote, On 4/18/2007 12:54 PM: > I've made some changes, uncommitted, that let you write something like > this: > > === > type file {}; > > (file i) p() { > app { > echo "hi" stdout=@filename(i); > } > } > > file q <"volume.blah">; > file r ; > > q=p(); > r=p(); > > === > > This will create two files, once called "volume.blah" and one called > "foovolume.blah". > > The point of this is to support the use cases from Tibi and Nika that > motivated bug 24, one of the 0.2 target features. > > So now I guess I should see if this functionality is actually helpful for > them in terms of expressing those workflows more concisely... > -- Mike Wilde Computation Institute, University of Chicago Math & Computer Science Division Argonne National Laboratory Argonne, IL 60439 USA tel 630-252-7497 fax 630-252-1997 From benc at hawaga.org.uk Thu Apr 19 02:43:52 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 19 Apr 2007 07:43:52 +0000 (GMT) Subject: [Swift-devel] strcat In-Reply-To: <46265F71.7000100@mcs.anl.gov> References: <46265F71.7000100@mcs.anl.gov> Message-ID: On Wed, 18 Apr 2007, Mike Wilde wrote: > If this sounds reasonable, then Nika and Tibi, could you each set up one > simple but working example for Ben to try, on terminable? Tibi had some stuff on teraport already, so I played with that a little bit and came up with some further work to do. -- From nefedova at mcs.anl.gov Tue Apr 24 10:48:11 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 10:48:11 -0500 Subject: [Swift-devel] nightly build error? Message-ID: <6.0.0.22.2.20070424104620.0332fec0@mail.mcs.anl.gov> Hi, It seems like there is an error in the latest nightly build (070423): [nefedova at terminable fermi_bench]$ swift lqcd-genU.dtm /home/nefedova/vdsk-070423/bin/swift: line 85: java: command not found (I really would like to use this version since it has "strcat") Thanks, Nika From hategan at mcs.anl.gov Tue Apr 24 10:50:07 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 24 Apr 2007 10:50:07 -0500 Subject: [Swift-devel] nightly build error? In-Reply-To: <6.0.0.22.2.20070424104620.0332fec0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070424104620.0332fec0@mail.mcs.anl.gov> Message-ID: <1177429808.2413.0.camel@blabla.mcs.anl.gov> On Tue, 2007-04-24 at 10:48 -0500, Veronika V. Nefedova wrote: > Hi, > > It seems like there is an error in the latest nightly build (070423): > > [nefedova at terminable fermi_bench]$ swift lqcd-genU.dtm > /home/nefedova/vdsk-070423/bin/swift: line 85: *****java: command not found***** Perhaps java not in the path? > > (I really would like to use this version since it has "strcat") > > Thanks, > > Nika > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From nefedova at mcs.anl.gov Tue Apr 24 11:06:53 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 11:06:53 -0500 Subject: [Swift-devel] nightly build error? In-Reply-To: <1177429808.2413.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424104620.0332fec0@mail.mcs.anl.gov> <1177429808.2413.0.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070424110344.04ee0870@mail.mcs.anl.gov> Grrr... That why I didn't want to run things on terminable in the first place! I had java in my path yesterday (and was running swift just fine), but today it disappeared. This is my soft file: @default +apache-ant-1.6.5-r1 +gx-map +condor +gx-map @globus-4 @default +matlab +R Am I missing anything? Nika At 10:50 AM 4/24/2007, Mihael Hategan wrote: >On Tue, 2007-04-24 at 10:48 -0500, Veronika V. Nefedova wrote: > > Hi, > > > > It seems like there is an error in the latest nightly build (070423): > > > > [nefedova at terminable fermi_bench]$ swift lqcd-genU.dtm > > /home/nefedova/vdsk-070423/bin/swift: line 85: *****java: command not > found***** > >Perhaps java not in the path? > > > > > (I really would like to use this version since it has "strcat") > > > > Thanks, > > > > Nika > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From nefedova at mcs.anl.gov Tue Apr 24 13:23:41 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 13:23:41 -0500 Subject: [Swift-devel] file declaration inside foreach Message-ID: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> Hi, I am wondering why this is not working? foreach i in range { file in References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> Message-ID: <1177439552.10659.0.camel@blabla.mcs.anl.gov> Posting the kml file would be useful. Mihael On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > Hi, > > I am wondering why this is not working? > > foreach i in range { > file in file out out=lqcd_exec(original,in); > } > > Swift complains about file declarations (mappers) inside foreach statement: > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > Could not compile SwiftScript source: line 11:1: unexpected token: file > [nefedova at evitable fermi_bench]$ > > Please let me know how to make it working (-; > > Thanks! > > Nika > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Tue Apr 24 13:39:13 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 24 Apr 2007 13:39:13 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177439552.10659.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> Message-ID: <1177439953.10946.0.camel@blabla.mcs.anl.gov> On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > Posting the kml file would be useful. Riiight. The nonexistent kml file. Nevermind me. > > Mihael > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > Hi, > > > > I am wondering why this is not working? > > > > foreach i in range { > > file in > file out > out=lqcd_exec(original,in); > > } > > > > Swift complains about file declarations (mappers) inside foreach statement: > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > Could not compile SwiftScript source: line 11:1: unexpected token: file > > [nefedova at evitable fermi_bench]$ > > > > Please let me know how to make it working (-; > > > > Thanks! > > > > Nika > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From nefedova at mcs.anl.gov Tue Apr 24 13:43:40 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 13:43:40 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177439552.10659.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070424134111.052b90d0@mail.mcs.anl.gov> All my files are on evitable in ~nefedova/LQCD/fermi_bench/lqcd-genU.* But I do not see any kml file, only xml and dtm Nika At 01:32 PM 4/24/2007, Mihael Hategan wrote: >Posting the kml file would be useful. > >Mihael > >On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > Hi, > > > > I am wondering why this is not working? > > > > foreach i in range { > > file in > file out > out=lqcd_exec(original,in); > > } > > > > Swift complains about file declarations (mappers) inside foreach statement: > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > Could not compile SwiftScript source: line 11:1: unexpected token: file > > [nefedova at evitable fermi_bench]$ > > > > Please let me know how to make it working (-; > > > > Thanks! > > > > Nika > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From yongzh at cs.uchicago.edu Tue Apr 24 14:49:02 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Tue, 24 Apr 2007 14:49:02 -0500 (CDT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> Message-ID: Nika, you are missing the ending '>' for both declaration. Yong. On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > Hi, > > I am wondering why this is not working? > > foreach i in range { > file in file out out=lqcd_exec(original,in); > } > > Swift complains about file declarations (mappers) inside foreach statement: > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > Could not compile SwiftScript source: line 11:1: unexpected token: file > [nefedova at evitable fermi_bench]$ > > Please let me know how to make it working (-; > > Thanks! > > Nika > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From nefedova at mcs.anl.gov Tue Apr 24 14:54:37 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 14:54:37 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177439953.10946.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> I did some modifications to the dtm file, and now i get something different: [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm Could not start execution. Failed to convert .xml to .kml for lqcd-genU.dtm [nefedova at evitable fermi_bench]$ pwd /autonfs/home/nefedova/LQCD/fermi_bench [nefedova at evitable fermi_bench]$ Any idea what the problem could be? Thanks, Nika At 01:39 PM 4/24/2007, Mihael Hategan wrote: >On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > Posting the kml file would be useful. > >Riiight. The nonexistent kml file. Nevermind me. > > > > > Mihael > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > Hi, > > > > > > I am wondering why this is not working? > > > > > > foreach i in range { > > > file in > > file out > > out=lqcd_exec(original,in); > > > } > > > > > > Swift complains about file declarations (mappers) inside foreach > statement: > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > Could not compile SwiftScript source: line 11:1: unexpected token: file > > > [nefedova at evitable fermi_bench]$ > > > > > > Please let me know how to make it working (-; > > > > > > Thanks! > > > > > > Nika > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From nefedova at mcs.anl.gov Tue Apr 24 14:54:55 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 14:54:55 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070424145441.052ca060@mail.mcs.anl.gov> right, I noticed that too ); At 02:49 PM 4/24/2007, Yong Zhao wrote: >Nika, > >you are missing the ending '>' for both declaration. > >Yong. > >On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > Hi, > > > > I am wondering why this is not working? > > > > foreach i in range { > > file in > file out > out=lqcd_exec(original,in); > > } > > > > Swift complains about file declarations (mappers) inside foreach statement: > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > Could not compile SwiftScript source: line 11:1: unexpected token: file > > [nefedova at evitable fermi_bench]$ > > > > Please let me know how to make it working (-; > > > > Thanks! > > > > Nika > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From benc at hawaga.org.uk Tue Apr 24 15:11:29 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 24 Apr 2007 20:11:29 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> Message-ID: run with -debug and you should get a much longer output. On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > I did some modifications to the dtm file, and now i get something different: > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > Could not start execution. > Failed to convert .xml to .kml for lqcd-genU.dtm > [nefedova at evitable fermi_bench]$ pwd > /autonfs/home/nefedova/LQCD/fermi_bench > [nefedova at evitable fermi_bench]$ > > Any idea what the problem could be? > > Thanks, > > Nika > > At 01:39 PM 4/24/2007, Mihael Hategan wrote: > > On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > > Posting the kml file would be useful. > > > > Riiight. The nonexistent kml file. Nevermind me. > > > > > > > > Mihael > > > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > > Hi, > > > > > > > > I am wondering why this is not working? > > > > > > > > foreach i in range { > > > > file in > > > file out > > > out=lqcd_exec(original,in); > > > > } > > > > > > > > Swift complains about file declarations (mappers) inside foreach > > statement: > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > Could not compile SwiftScript source: line 11:1: unexpected token: file > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > Please let me know how to make it working (-; > > > > > > > > Thanks! > > > > > > > > Nika > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From nefedova at mcs.anl.gov Tue Apr 24 15:15:51 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 15:15:51 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> You right! here is the output: [nefedova at evitable fermi_bench]$ swift -debug lqcd-genU.dtm lqcd-genU.dtm: source file is new. Recompiling. Detailed exception: java.lang.RuntimeException: Failed to convert .xml to .kml for lqcd-genU.dtm at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown Source) at org.griphyn.vdl.karajan.Loader.main(java.lang.String[]) (Unknown Source) at gnu.java.lang.MainThread.call_main() (/usr/lib/libgcj.so.6.0.0) at gnu.java.lang.MainThread.run() (/usr/lib/libgcj.so.6.0.0) Caused by: java.lang.NullPointerException at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(java.io.File, org.apache.xmlbeans.SchemaType, org.apache.xmlbeans.XmlOptions) (Unknown Source) at org.griphyn.vdl.model.ProgramDocument$Factory.parse(java.io.File) (Unknown Source) at org.griphyn.vdl.engine.Karajan.main(java.lang.String[]) (Unknown Source) at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown Source) ...3 more Could not start execution. Failed to convert .xml to .kml for lqcd-genU.dtm [nefedova at evitable fermi_bench]$ Nika At 03:11 PM 4/24/2007, Ben Clifford wrote: >run with -debug and you should get a much longer output. > >On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > I did some modifications to the dtm file, and now i get something > different: > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > Could not start execution. > > Failed to convert .xml to .kml for lqcd-genU.dtm > > [nefedova at evitable fermi_bench]$ pwd > > /autonfs/home/nefedova/LQCD/fermi_bench > > [nefedova at evitable fermi_bench]$ > > > > Any idea what the problem could be? > > > > Thanks, > > > > Nika > > > > At 01:39 PM 4/24/2007, Mihael Hategan wrote: > > > On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > > > Posting the kml file would be useful. > > > > > > Riiight. The nonexistent kml file. Nevermind me. > > > > > > > > > > > Mihael > > > > > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > > > Hi, > > > > > > > > > > I am wondering why this is not working? > > > > > > > > > > foreach i in range { > > > > > file in > > > > file out > > > > out=lqcd_exec(original,in); > > > > > } > > > > > > > > > > Swift complains about file declarations (mappers) inside foreach > > > statement: > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > Could not compile SwiftScript source: line 11:1: unexpected > token: file > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > Please let me know how to make it working (-; > > > > > > > > > > Thanks! > > > > > > > > > > Nika > > > > > > > > > > _______________________________________________ > > > > > Swift-devel mailing list > > > > > Swift-devel at ci.uchicago.edu > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > From benc at hawaga.org.uk Tue Apr 24 15:17:43 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 24 Apr 2007 20:17:43 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> Message-ID: GCJ! that isn't normal java. is that what you normally use? it used to be lame, though i have no idea now. also evitable is unsupported. try terminable. On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > You right! here is the output: > > [nefedova at evitable fermi_bench]$ swift -debug lqcd-genU.dtm > lqcd-genU.dtm: source file is new. Recompiling. > Detailed exception: > java.lang.RuntimeException: Failed to convert .xml to .kml for lqcd-genU.dtm > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > Source) > at org.griphyn.vdl.karajan.Loader.main(java.lang.String[]) (Unknown Source) > at gnu.java.lang.MainThread.call_main() (/usr/lib/libgcj.so.6.0.0) > at gnu.java.lang.MainThread.run() (/usr/lib/libgcj.so.6.0.0) > Caused by: java.lang.NullPointerException > at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(java.io.File, > org.apache.xmlbeans.SchemaType, org.apache.xmlbeans.XmlOptions) (Unknown > Source) > at org.griphyn.vdl.model.ProgramDocument$Factory.parse(java.io.File) > (Unknown Source) > at org.griphyn.vdl.engine.Karajan.main(java.lang.String[]) (Unknown Source) > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > Source) > ...3 more > > Could not start execution. > Failed to convert .xml to .kml for lqcd-genU.dtm > [nefedova at evitable fermi_bench]$ > > Nika > > At 03:11 PM 4/24/2007, Ben Clifford wrote: > > > run with -debug and you should get a much longer output. > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > I did some modifications to the dtm file, and now i get something > > different: > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > Could not start execution. > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > [nefedova at evitable fermi_bench]$ pwd > > > /autonfs/home/nefedova/LQCD/fermi_bench > > > [nefedova at evitable fermi_bench]$ > > > > > > Any idea what the problem could be? > > > > > > Thanks, > > > > > > Nika > > > > > > At 01:39 PM 4/24/2007, Mihael Hategan wrote: > > > > On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > > > > Posting the kml file would be useful. > > > > > > > > Riiight. The nonexistent kml file. Nevermind me. > > > > > > > > > > > > > > Mihael > > > > > > > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > > > > Hi, > > > > > > > > > > > > I am wondering why this is not working? > > > > > > > > > > > > foreach i in range { > > > > > > file in > > > > > file out > > > > > out=lqcd_exec(original,in); > > > > > > } > > > > > > > > > > > > Swift complains about file declarations (mappers) inside foreach > > > > statement: > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > Could not compile SwiftScript source: line 11:1: unexpected token: > > file > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > Please let me know how to make it working (-; > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Nika > > > > > > > > > > > > _______________________________________________ > > > > > > Swift-devel mailing list > > > > > > Swift-devel at ci.uchicago.edu > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Swift-devel mailing list > > > > > Swift-devel at ci.uchicago.edu > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > From nefedova at mcs.anl.gov Tue Apr 24 15:37:24 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 15:37:24 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> Fortunately, Ti was able to fix my env on terminable (some faulty softenv key thats not working in tcsh) so I was able to try the run there. The compilation was fine, no problems. Then it submitted the workflow. The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). And since the jobs were supposed to run one after another - they all failed (application failure) Nika At 03:17 PM 4/24/2007, Ben Clifford wrote: >GCJ! that isn't normal java. is that what you normally use? it used to be >lame, though i have no idea now. also evitable is unsupported. try >terminable. > >On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > You right! here is the output: > > > > [nefedova at evitable fermi_bench]$ swift -debug lqcd-genU.dtm > > lqcd-genU.dtm: source file is new. Recompiling. > > Detailed exception: > > java.lang.RuntimeException: Failed to convert .xml to .kml for > lqcd-genU.dtm > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > Source) > > at org.griphyn.vdl.karajan.Loader.main(java.lang.String[]) (Unknown > Source) > > at gnu.java.lang.MainThread.call_main() (/usr/lib/libgcj.so.6.0.0) > > at gnu.java.lang.MainThread.run() (/usr/lib/libgcj.so.6.0.0) > > Caused by: java.lang.NullPointerException > > at > org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(java.io.File, > > org.apache.xmlbeans.SchemaType, org.apache.xmlbeans.XmlOptions) (Unknown > > Source) > > at org.griphyn.vdl.model.ProgramDocument$Factory.parse(java.io.File) > > (Unknown Source) > > at org.griphyn.vdl.engine.Karajan.main(java.lang.String[]) (Unknown > Source) > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > Source) > > ...3 more > > > > Could not start execution. > > Failed to convert .xml to .kml for lqcd-genU.dtm > > [nefedova at evitable fermi_bench]$ > > > > Nika > > > > At 03:11 PM 4/24/2007, Ben Clifford wrote: > > > > > run with -debug and you should get a much longer output. > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > I did some modifications to the dtm file, and now i get something > > > different: > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > Could not start execution. > > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > > [nefedova at evitable fermi_bench]$ pwd > > > > /autonfs/home/nefedova/LQCD/fermi_bench > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > Any idea what the problem could be? > > > > > > > > Thanks, > > > > > > > > Nika > > > > > > > > At 01:39 PM 4/24/2007, Mihael Hategan wrote: > > > > > On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > > > > > Posting the kml file would be useful. > > > > > > > > > > Riiight. The nonexistent kml file. Nevermind me. > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > > > > > Hi, > > > > > > > > > > > > > > I am wondering why this is not working? > > > > > > > > > > > > > > foreach i in range { > > > > > > > file in > > > > > > file out > > > > > > out=lqcd_exec(original,in); > > > > > > > } > > > > > > > > > > > > > > Swift complains about file declarations (mappers) inside foreach > > > > > statement: > > > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > > Could not compile SwiftScript source: line 11:1: unexpected > token: > > > file > > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > > > Please let me know how to make it working (-; > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Swift-devel mailing list > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Swift-devel mailing list > > > > > > Swift-devel at ci.uchicago.edu > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > From benc at hawaga.org.uk Tue Apr 24 15:44:59 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 24 Apr 2007 20:44:59 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> Message-ID: On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). And > since the jobs were supposed to run one after another - they all failed > (application failure) ok. That's expected, or at least, I expect it. foreach is a parallel map-like operation for running some action on each element in an array independent of the other elements. I think you probably can't express that kind of iteration in SwiftScript at the moment (unless 'while' happens to work for you) though I'm poking round with ways to express that. (in functional programming terms, you're trying to do a fold rather than a map, I think). -- From hategan at mcs.anl.gov Tue Apr 24 15:47:12 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 24 Apr 2007 15:47:12 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> Message-ID: <1177447633.15483.3.camel@blabla.mcs.anl.gov> Because the jobs are independent. You are creating iteratiion-local variables between which there is no dependency. What you want is something like this: file[] lattice <...>; lattice[0] = initialLattice; for i in 1...2 { lattice[i] = app(lattice[i-1]); } On Tue, 2007-04-24 at 15:37 -0500, Veronika V. Nefedova wrote: > Fortunately, Ti was able to fix my env on terminable (some faulty softenv > key thats not working in tcsh) so I was able to try the run there. > The compilation was fine, no problems. Then it submitted the workflow. > > The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). > And since the jobs were supposed to run one after another - they all failed > (application failure) > > Nika > > > > At 03:17 PM 4/24/2007, Ben Clifford wrote: > > >GCJ! that isn't normal java. is that what you normally use? it used to be > >lame, though i have no idea now. also evitable is unsupported. try > >terminable. > > > >On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > You right! here is the output: > > > > > > [nefedova at evitable fermi_bench]$ swift -debug lqcd-genU.dtm > > > lqcd-genU.dtm: source file is new. Recompiling. > > > Detailed exception: > > > java.lang.RuntimeException: Failed to convert .xml to .kml for > > lqcd-genU.dtm > > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > > Source) > > > at org.griphyn.vdl.karajan.Loader.main(java.lang.String[]) (Unknown > > Source) > > > at gnu.java.lang.MainThread.call_main() (/usr/lib/libgcj.so.6.0.0) > > > at gnu.java.lang.MainThread.run() (/usr/lib/libgcj.so.6.0.0) > > > Caused by: java.lang.NullPointerException > > > at > > org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(java.io.File, > > > org.apache.xmlbeans.SchemaType, org.apache.xmlbeans.XmlOptions) (Unknown > > > Source) > > > at org.griphyn.vdl.model.ProgramDocument$Factory.parse(java.io.File) > > > (Unknown Source) > > > at org.griphyn.vdl.engine.Karajan.main(java.lang.String[]) (Unknown > > Source) > > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > > Source) > > > ...3 more > > > > > > Could not start execution. > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > [nefedova at evitable fermi_bench]$ > > > > > > Nika > > > > > > At 03:11 PM 4/24/2007, Ben Clifford wrote: > > > > > > > run with -debug and you should get a much longer output. > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > I did some modifications to the dtm file, and now i get something > > > > different: > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > Could not start execution. > > > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > > > [nefedova at evitable fermi_bench]$ pwd > > > > > /autonfs/home/nefedova/LQCD/fermi_bench > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > Any idea what the problem could be? > > > > > > > > > > Thanks, > > > > > > > > > > Nika > > > > > > > > > > At 01:39 PM 4/24/2007, Mihael Hategan wrote: > > > > > > On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > > > > > > Posting the kml file would be useful. > > > > > > > > > > > > Riiight. The nonexistent kml file. Nevermind me. > > > > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > I am wondering why this is not working? > > > > > > > > > > > > > > > > foreach i in range { > > > > > > > > file in > > > > > > > file out > > > > > > > out=lqcd_exec(original,in); > > > > > > > > } > > > > > > > > > > > > > > > > Swift complains about file declarations (mappers) inside foreach > > > > > > statement: > > > > > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > > > Could not compile SwiftScript source: line 11:1: unexpected > > token: > > > > file > > > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > > > > > Please let me know how to make it working (-; > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > Swift-devel mailing list > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Swift-devel mailing list > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Swift-devel mailing list > > > > > Swift-devel at ci.uchicago.edu > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > From benc at hawaga.org.uk Tue Apr 24 16:00:11 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 24 Apr 2007 21:00:11 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177447633.15483.3.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> Message-ID: there isn't a plain for statement at the moment. but it used to look like a C/java style for, not a range-based for - like: for(i=1; i<2; i++) { ... } which doesn't work given single assignment variables. On Tue, 24 Apr 2007, Mihael Hategan wrote: > Because the jobs are independent. You are creating iteratiion-local > variables between which there is no dependency. > > What you want is something like this: > file[] lattice <...>; > lattice[0] = initialLattice; > for i in 1...2 { > lattice[i] = app(lattice[i-1]); > } > > On Tue, 2007-04-24 at 15:37 -0500, Veronika V. Nefedova wrote: > > Fortunately, Ti was able to fix my env on terminable (some faulty softenv > > key thats not working in tcsh) so I was able to try the run there. > > The compilation was fine, no problems. Then it submitted the workflow. > > > > The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). > > And since the jobs were supposed to run one after another - they all failed > > (application failure) > > > > Nika > > > > > > > > At 03:17 PM 4/24/2007, Ben Clifford wrote: > > > > >GCJ! that isn't normal java. is that what you normally use? it used to be > > >lame, though i have no idea now. also evitable is unsupported. try > > >terminable. > > > > > >On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > You right! here is the output: > > > > > > > > [nefedova at evitable fermi_bench]$ swift -debug lqcd-genU.dtm > > > > lqcd-genU.dtm: source file is new. Recompiling. > > > > Detailed exception: > > > > java.lang.RuntimeException: Failed to convert .xml to .kml for > > > lqcd-genU.dtm > > > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > > > Source) > > > > at org.griphyn.vdl.karajan.Loader.main(java.lang.String[]) (Unknown > > > Source) > > > > at gnu.java.lang.MainThread.call_main() (/usr/lib/libgcj.so.6.0.0) > > > > at gnu.java.lang.MainThread.run() (/usr/lib/libgcj.so.6.0.0) > > > > Caused by: java.lang.NullPointerException > > > > at > > > org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(java.io.File, > > > > org.apache.xmlbeans.SchemaType, org.apache.xmlbeans.XmlOptions) (Unknown > > > > Source) > > > > at org.griphyn.vdl.model.ProgramDocument$Factory.parse(java.io.File) > > > > (Unknown Source) > > > > at org.griphyn.vdl.engine.Karajan.main(java.lang.String[]) (Unknown > > > Source) > > > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > > > Source) > > > > ...3 more > > > > > > > > Could not start execution. > > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > Nika > > > > > > > > At 03:11 PM 4/24/2007, Ben Clifford wrote: > > > > > > > > > run with -debug and you should get a much longer output. > > > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > > > I did some modifications to the dtm file, and now i get something > > > > > different: > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > Could not start execution. > > > > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > > > > [nefedova at evitable fermi_bench]$ pwd > > > > > > /autonfs/home/nefedova/LQCD/fermi_bench > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > Any idea what the problem could be? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Nika > > > > > > > > > > > > At 01:39 PM 4/24/2007, Mihael Hategan wrote: > > > > > > > On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > > > > > > > Posting the kml file would be useful. > > > > > > > > > > > > > > Riiight. The nonexistent kml file. Nevermind me. > > > > > > > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I am wondering why this is not working? > > > > > > > > > > > > > > > > > > foreach i in range { > > > > > > > > > file in > > > > > > > > file out > > > > > > > > out=lqcd_exec(original,in); > > > > > > > > > } > > > > > > > > > > > > > > > > > > Swift complains about file declarations (mappers) inside foreach > > > > > > > statement: > > > > > > > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > > > > Could not compile SwiftScript source: line 11:1: unexpected > > > token: > > > > > file > > > > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > > > > > > > Please let me know how to make it working (-; > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > Swift-devel mailing list > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > Swift-devel mailing list > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Swift-devel mailing list > > > > > > Swift-devel at ci.uchicago.edu > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hategan at mcs.anl.gov Tue Apr 24 16:09:33 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 24 Apr 2007 16:09:33 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> Message-ID: <1177448973.16453.1.camel@blabla.mcs.anl.gov> It's been hacked around for a while now: int[] r = [1:2:1]; for i in r { } I would though like to see for i in 1..2 working though. On Tue, 2007-04-24 at 21:00 +0000, Ben Clifford wrote: > there isn't a plain for statement at the moment. but it used to look like > a C/java style for, not a range-based for - like: > > for(i=1; i<2; i++) { > ... > } > > which doesn't work given single assignment variables. > > On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > Because the jobs are independent. You are creating iteratiion-local > > variables between which there is no dependency. > > > > What you want is something like this: > > file[] lattice <...>; > > lattice[0] = initialLattice; > > for i in 1...2 { > > lattice[i] = app(lattice[i-1]); > > } > > > > On Tue, 2007-04-24 at 15:37 -0500, Veronika V. Nefedova wrote: > > > Fortunately, Ti was able to fix my env on terminable (some faulty softenv > > > key thats not working in tcsh) so I was able to try the run there. > > > The compilation was fine, no problems. Then it submitted the workflow. > > > > > > The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). > > > And since the jobs were supposed to run one after another - they all failed > > > (application failure) > > > > > > Nika > > > > > > > > > > > > At 03:17 PM 4/24/2007, Ben Clifford wrote: > > > > > > >GCJ! that isn't normal java. is that what you normally use? it used to be > > > >lame, though i have no idea now. also evitable is unsupported. try > > > >terminable. > > > > > > > >On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > You right! here is the output: > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift -debug lqcd-genU.dtm > > > > > lqcd-genU.dtm: source file is new. Recompiling. > > > > > Detailed exception: > > > > > java.lang.RuntimeException: Failed to convert .xml to .kml for > > > > lqcd-genU.dtm > > > > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > > > > Source) > > > > > at org.griphyn.vdl.karajan.Loader.main(java.lang.String[]) (Unknown > > > > Source) > > > > > at gnu.java.lang.MainThread.call_main() (/usr/lib/libgcj.so.6.0.0) > > > > > at gnu.java.lang.MainThread.run() (/usr/lib/libgcj.so.6.0.0) > > > > > Caused by: java.lang.NullPointerException > > > > > at > > > > org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(java.io.File, > > > > > org.apache.xmlbeans.SchemaType, org.apache.xmlbeans.XmlOptions) (Unknown > > > > > Source) > > > > > at org.griphyn.vdl.model.ProgramDocument$Factory.parse(java.io.File) > > > > > (Unknown Source) > > > > > at org.griphyn.vdl.engine.Karajan.main(java.lang.String[]) (Unknown > > > > Source) > > > > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > > > > Source) > > > > > ...3 more > > > > > > > > > > Could not start execution. > > > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > Nika > > > > > > > > > > At 03:11 PM 4/24/2007, Ben Clifford wrote: > > > > > > > > > > > run with -debug and you should get a much longer output. > > > > > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > > > > > I did some modifications to the dtm file, and now i get something > > > > > > different: > > > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > > Could not start execution. > > > > > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > > > > > [nefedova at evitable fermi_bench]$ pwd > > > > > > > /autonfs/home/nefedova/LQCD/fermi_bench > > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > > > Any idea what the problem could be? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > At 01:39 PM 4/24/2007, Mihael Hategan wrote: > > > > > > > > On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > > > > > > > > Posting the kml file would be useful. > > > > > > > > > > > > > > > > Riiight. The nonexistent kml file. Nevermind me. > > > > > > > > > > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > I am wondering why this is not working? > > > > > > > > > > > > > > > > > > > > foreach i in range { > > > > > > > > > > file in > > > > > > > > > file out > > > > > > > > > out=lqcd_exec(original,in); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > Swift complains about file declarations (mappers) inside foreach > > > > > > > > statement: > > > > > > > > > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > > > > > Could not compile SwiftScript source: line 11:1: unexpected > > > > token: > > > > > > file > > > > > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > > > > > > > > > Please let me know how to make it working (-; > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > Swift-devel mailing list > > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > Swift-devel mailing list > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Swift-devel mailing list > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From yongzh at cs.uchicago.edu Tue Apr 24 16:30:49 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Tue, 24 Apr 2007 16:30:49 -0500 (CDT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> Message-ID: There was a sequential option for 'foreach', and maybe we can translate that into a sequential for in karajan. Yong. On Tue, 24 Apr 2007, Ben Clifford wrote: > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). And > > since the jobs were supposed to run one after another - they all failed > > (application failure) > > ok. That's expected, or at least, I expect it. > > foreach is a parallel map-like operation for running some action on each > element in an array independent of the other elements. > > I think you probably can't express that kind of iteration in SwiftScript > at the moment (unless 'while' happens to work for you) though I'm poking > round with ways to express that. (in functional programming terms, you're > trying to do a fold rather than a map, I think). > > -- > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Tue Apr 24 16:33:30 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 24 Apr 2007 16:33:30 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> Message-ID: <1177450410.17344.0.camel@blabla.mcs.anl.gov> Why if we can actually express the problem? On Tue, 2007-04-24 at 16:30 -0500, Yong Zhao wrote: > There was a sequential option for 'foreach', and maybe we can translate > that into a sequential for in karajan. > > Yong. > > On Tue, 24 Apr 2007, Ben Clifford wrote: > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). And > > > since the jobs were supposed to run one after another - they all failed > > > (application failure) > > > > ok. That's expected, or at least, I expect it. > > > > foreach is a parallel map-like operation for running some action on each > > element in an array independent of the other elements. > > > > I think you probably can't express that kind of iteration in SwiftScript > > at the moment (unless 'while' happens to work for you) though I'm poking > > round with ways to express that. (in functional programming terms, you're > > trying to do a fold rather than a map, I think). > > > > -- > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From yongzh at cs.uchicago.edu Tue Apr 24 16:45:00 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Tue, 24 Apr 2007 16:45:00 -0500 (CDT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177450410.17344.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177450410.17344.0.camel@blabla.mcs.anl.gov> Message-ID: Right, if there is data dependency, then it is not necessary. Yong. On Tue, 24 Apr 2007, Mihael Hategan wrote: > Why if we can actually express the problem? > > On Tue, 2007-04-24 at 16:30 -0500, Yong Zhao wrote: > > There was a sequential option for 'foreach', and maybe we can translate > > that into a sequential for in karajan. > > > > Yong. > > > > On Tue, 24 Apr 2007, Ben Clifford wrote: > > > > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). And > > > > since the jobs were supposed to run one after another - they all failed > > > > (application failure) > > > > > > ok. That's expected, or at least, I expect it. > > > > > > foreach is a parallel map-like operation for running some action on each > > > element in an array independent of the other elements. > > > > > > I think you probably can't express that kind of iteration in SwiftScript > > > at the moment (unless 'while' happens to work for you) though I'm poking > > > round with ways to express that. (in functional programming terms, you're > > > trying to do a fold rather than a map, I think). > > > > > > -- > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > From nefedova at mcs.anl.gov Tue Apr 24 16:53:57 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 16:53:57 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177450410.17344.0.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070424164910.0618a490@mail.mcs.anl.gov> Well, I also need to run one small program inside that loop that doesn't have that data dependency. Basically what that program does is modifies the input file for the main application at every loop step: original=modify(original,i); I am sure this one wouldn't work (; and I'd have to create an array of input files (one for each i) ? Nika At 04:45 PM 4/24/2007, Yong Zhao wrote: >Right, if there is data dependency, then it is not necessary. > >Yong. > >On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > Why if we can actually express the problem? > > > > On Tue, 2007-04-24 at 16:30 -0500, Yong Zhao wrote: > > > There was a sequential option for 'foreach', and maybe we can translate > > > that into a sequential for in karajan. > > > > > > Yong. > > > > > > On Tue, 24 Apr 2007, Ben Clifford wrote: > > > > > > > > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > The main problem: it submitted 2 jobs at once (foreach loop is > for i=1,2). And > > > > > since the jobs were supposed to run one after another - they all > failed > > > > > (application failure) > > > > > > > > ok. That's expected, or at least, I expect it. > > > > > > > > foreach is a parallel map-like operation for running some action on > each > > > > element in an array independent of the other elements. > > > > > > > > I think you probably can't express that kind of iteration in > SwiftScript > > > > at the moment (unless 'while' happens to work for you) though I'm > poking > > > > round with ways to express that. (in functional programming terms, > you're > > > > trying to do a fold rather than a map, I think). > > > > > > > > -- > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Tue Apr 24 17:15:18 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 24 Apr 2007 17:15:18 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <6.0.0.22.2.20070424164910.0618a490@mail.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177450410.17344.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424164910.0618a490@mail.mcs.anl.gov> Message-ID: <1177452918.19533.2.camel@blabla.mcs.anl.gov> So which is it? Does each step depend on the previous step, or some modification of the initial data, or both? On Tue, 2007-04-24 at 16:53 -0500, Veronika V. Nefedova wrote: > Well, I also need to run one small program inside that loop that doesn't > have that data dependency. Basically what that program does is modifies the > input file for the main application at every loop step: > > original=modify(original,i); > > I am sure this one wouldn't work (; and I'd have to create an array of > input files (one for each i) ? > > Nika > > At 04:45 PM 4/24/2007, Yong Zhao wrote: > >Right, if there is data dependency, then it is not necessary. > > > >Yong. > > > >On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > > > Why if we can actually express the problem? > > > > > > On Tue, 2007-04-24 at 16:30 -0500, Yong Zhao wrote: > > > > There was a sequential option for 'foreach', and maybe we can translate > > > > that into a sequential for in karajan. > > > > > > > > Yong. > > > > > > > > On Tue, 24 Apr 2007, Ben Clifford wrote: > > > > > > > > > > > > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > > > The main problem: it submitted 2 jobs at once (foreach loop is > > for i=1,2). And > > > > > > since the jobs were supposed to run one after another - they all > > failed > > > > > > (application failure) > > > > > > > > > > ok. That's expected, or at least, I expect it. > > > > > > > > > > foreach is a parallel map-like operation for running some action on > > each > > > > > element in an array independent of the other elements. > > > > > > > > > > I think you probably can't express that kind of iteration in > > SwiftScript > > > > > at the moment (unless 'while' happens to work for you) though I'm > > poking > > > > > round with ways to express that. (in functional programming terms, > > you're > > > > > trying to do a fold rather than a map, I think). > > > > > > > > > > -- > > > > > _______________________________________________ > > > > > Swift-devel mailing list > > > > > Swift-devel at ci.uchicago.edu > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > >_______________________________________________ > >Swift-devel mailing list > >Swift-devel at ci.uchicago.edu > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From nefedova at mcs.anl.gov Tue Apr 24 17:23:57 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Tue, 24 Apr 2007 17:23:57 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177452918.19533.2.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177450410.17344.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424164910.0618a490@mail.mcs.anl.gov> <1177452918.19533.2.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070424171913.052c76b0@mail.mcs.anl.gov> nope, each modification is independent of each other (you could modify the file from the previous step, or from the step 0), but we need to make sure that the correct input file is present for each main application run (in the loop). I guess an array would work? (; Nika At 05:15 PM 4/24/2007, Mihael Hategan wrote: >So which is it? Does each step depend on the previous step, or some >modification of the initial data, or both? > >On Tue, 2007-04-24 at 16:53 -0500, Veronika V. Nefedova wrote: > > Well, I also need to run one small program inside that loop that doesn't > > have that data dependency. Basically what that program does is modifies > the > > input file for the main application at every loop step: > > > > original=modify(original,i); > > > > I am sure this one wouldn't work (; and I'd have to create an array of > > input files (one for each i) ? > > > > Nika > > > > At 04:45 PM 4/24/2007, Yong Zhao wrote: > > >Right, if there is data dependency, then it is not necessary. > > > > > >Yong. > > > > > >On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > > > > > Why if we can actually express the problem? > > > > > > > > On Tue, 2007-04-24 at 16:30 -0500, Yong Zhao wrote: > > > > > There was a sequential option for 'foreach', and maybe we can > translate > > > > > that into a sequential for in karajan. > > > > > > > > > > Yong. > > > > > > > > > > On Tue, 24 Apr 2007, Ben Clifford wrote: > > > > > > > > > > > > > > > > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > > > > > The main problem: it submitted 2 jobs at once (foreach loop is > > > for i=1,2). And > > > > > > > since the jobs were supposed to run one after another - they all > > > failed > > > > > > > (application failure) > > > > > > > > > > > > ok. That's expected, or at least, I expect it. > > > > > > > > > > > > foreach is a parallel map-like operation for running some > action on > > > each > > > > > > element in an array independent of the other elements. > > > > > > > > > > > > I think you probably can't express that kind of iteration in > > > SwiftScript > > > > > > at the moment (unless 'while' happens to work for you) though I'm > > > poking > > > > > > round with ways to express that. (in functional programming terms, > > > you're > > > > > > trying to do a fold rather than a map, I think). > > > > > > > > > > > > -- > > > > > > _______________________________________________ > > > > > > Swift-devel mailing list > > > > > > Swift-devel at ci.uchicago.edu > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > _______________________________________________ > > > > > Swift-devel mailing list > > > > > Swift-devel at ci.uchicago.edu > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > >_______________________________________________ > > >Swift-devel mailing list > > >Swift-devel at ci.uchicago.edu > > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > From hategan at mcs.anl.gov Tue Apr 24 17:45:27 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 24 Apr 2007 17:45:27 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <6.0.0.22.2.20070424171913.052c76b0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177450410.17344.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424164910.0618a490@mail.mcs.anl.gov> <1177452918.19533.2.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424171913.052c76b0@mail.mcs.anl.gov> Message-ID: <1177454727.20316.12.camel@blabla.mcs.anl.gov> If you modify the file from the previous step, then the two steps are not independent. An array would work, but you probably don't need it. foreach i in range { file p=modify(original, i); lattice[i] = lqcd_exec(p, lattice[i-1]); } On Tue, 2007-04-24 at 17:23 -0500, Veronika V. Nefedova wrote: > nope, each modification is independent of each other (you could modify the > file from the previous step, or from the step 0), but we need to make sure > that the correct input file is present for each main application run (in > the loop). I guess an array would work? (; > > Nika > > At 05:15 PM 4/24/2007, Mihael Hategan wrote: > >So which is it? Does each step depend on the previous step, or some > >modification of the initial data, or both? > > > >On Tue, 2007-04-24 at 16:53 -0500, Veronika V. Nefedova wrote: > > > Well, I also need to run one small program inside that loop that doesn't > > > have that data dependency. Basically what that program does is modifies > > the > > > input file for the main application at every loop step: > > > > > > original=modify(original,i); > > > > > > I am sure this one wouldn't work (; and I'd have to create an array of > > > input files (one for each i) ? > > > > > > Nika > > > > > > At 04:45 PM 4/24/2007, Yong Zhao wrote: > > > >Right, if there is data dependency, then it is not necessary. > > > > > > > >Yong. > > > > > > > >On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > > > > > > > Why if we can actually express the problem? > > > > > > > > > > On Tue, 2007-04-24 at 16:30 -0500, Yong Zhao wrote: > > > > > > There was a sequential option for 'foreach', and maybe we can > > translate > > > > > > that into a sequential for in karajan. > > > > > > > > > > > > Yong. > > > > > > > > > > > > On Tue, 24 Apr 2007, Ben Clifford wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > > > > > > > The main problem: it submitted 2 jobs at once (foreach loop is > > > > for i=1,2). And > > > > > > > > since the jobs were supposed to run one after another - they all > > > > failed > > > > > > > > (application failure) > > > > > > > > > > > > > > ok. That's expected, or at least, I expect it. > > > > > > > > > > > > > > foreach is a parallel map-like operation for running some > > action on > > > > each > > > > > > > element in an array independent of the other elements. > > > > > > > > > > > > > > I think you probably can't express that kind of iteration in > > > > SwiftScript > > > > > > > at the moment (unless 'while' happens to work for you) though I'm > > > > poking > > > > > > > round with ways to express that. (in functional programming terms, > > > > you're > > > > > > > trying to do a fold rather than a map, I think). > > > > > > > > > > > > > > -- > > > > > > > _______________________________________________ > > > > > > > Swift-devel mailing list > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > _______________________________________________ > > > > > > Swift-devel mailing list > > > > > > Swift-devel at ci.uchicago.edu > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > >_______________________________________________ > > > >Swift-devel mailing list > > > >Swift-devel at ci.uchicago.edu > > > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > From benc at hawaga.org.uk Wed Apr 25 01:26:31 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 06:26:31 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177448973.16453.1.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177448973.16453.1.camel@blabla.mcs.anl.gov> Message-ID: On Tue, 24 Apr 2007, Mihael Hategan wrote: > It's been hacked around for a while now: > int[] r = [1:2:1]; > for i in r { > } Do you mean foreach there? If not, then that doesn't correspond with my view of the history at all - the VDL.g grammar lost its production rule for for(;;) in r375, and immediately before that it looked like: "for" LPAREN e1=assignStat SEMI e2=expression SEMI e3=assignStat RPAREN which is for(a;b;c) syntax. -- From benc at hawaga.org.uk Wed Apr 25 01:30:35 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 06:30:35 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177450410.17344.0.camel@blabla.mcs.anl.gov> Message-ID: On Tue, 24 Apr 2007, Yong Zhao wrote: > Right, if there is data dependency, then it is not necessary. and if there isn't a data dependency, it shouldn't be necessary either... -- From benc at hawaga.org.uk Wed Apr 25 01:34:46 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 06:34:46 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177448973.16453.1.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177448973.16453.1.camel@blabla.mcs.anl.gov> Message-ID: On Tue, 24 Apr 2007, Mihael Hategan wrote: > I would though like to see for i in 1..2 working though. that should be relatively straight-forward syntactic sugar, but I don't particularly like introducing it as a special case for foreach - it should be a synonym everywhere for [1:2:1] if introduced. But bleugh, [1:2] is only one character more than 1..2 -- From benc at hawaga.org.uk Wed Apr 25 02:35:25 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 07:35:25 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177447633.15483.3.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> Message-ID: On Tue, 24 Apr 2007, Mihael Hategan wrote: > Because the jobs are independent. You are creating iteratiion-local > variables between which there is no dependency. > > What you want is something like this: > file[] lattice <...>; > lattice[0] = initialLattice; > for i in 1...2 { > lattice[i] = app(lattice[i-1]); > } I just tried this: int r[] = [1:2:1]; int k[]; k[0]=100; foreach i in r { k[i] = k[i-1] + i; } and I get this: $ swift for.swift Swift V 0.0405 RunID: 2hp99svgoyjd2 Execution failed: String index out of range: 0 :-( -- From benc at hawaga.org.uk Wed Apr 25 03:32:09 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 08:32:09 +0000 (GMT) Subject: [Swift-devel] Nasty multiparameter function hack. Message-ID: There's a nasty hack in the multiparameter function handling, which doesn't affect the extremely simple function invocations that we make at the moment but will whenever we try to do anything more advanced. Basically the problem I had was this: In the previous single-parameter syntax, a call like this: @f(3) or @f(1+2) (where the single parameter is a constant or an expression) turned into a Karajan function call with a single parameter. However, where the parameter is a variable: @f(x) this turned into a Karajan function call with two parameters, a 'variable' and a 'path'. In the above case, the variable is x and the path is empty. In the case of @f(x.y) then the variable would be x and the path would be y. This leads to inconsistencies, such as : x = 3 y = @f(x) resulting in a very different karajan function invocation to y = @f(3) This rather awkward arrangement is used by @filename because it essentially wants pass-by-reference rather than pass-by-value semantics in order to extract the filename of its passed parameter. I wanted to avoid indulging in massive expression evaluator hackery, so I put in the following behaviour for multi-parameter functions: For a function f with parameters x1, x2, x3, etc which may be constants or expressions or other function calls or variables (note that all three are treated differently), then: if x_n is a constant, expression or function call, we translate that into a single karajan-layer parameter. if x_n is a variable reference, then we translate that into two karajan parameters, first the path, and second the variable. This magically works for @strcat when used with top level variables, because path is empty, so @strcat("hi",x) translates into a concatenation of *three* things: "hi", the path "", and the contents of variable x, and as "" is a zero wrt @strcat then its the same as "hi" and the value of x. However, this approach will fail in at least the following circumstances (I think) i) @strcat("hi",x.y) (because now path will be non-null) ii) if we ever have a function that cares about the ordering of its parameters. I don't particularly propose resolving this for the 0.2 timeframe. However, this is a mess which *will* cause trouble later on, so a work item (perhaps for 0.3 milestone) should be to sort out this (and a number of other messy bits in the expression system). -- From benc at hawaga.org.uk Wed Apr 25 05:42:05 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 10:42:05 +0000 (GMT) Subject: [Swift-devel] Re: Nasty multiparameter function hack. In-Reply-To: References: Message-ID: On Wed, 25 Apr 2007, Ben Clifford wrote: > I don't particularly propose resolving this for the 0.2 timeframe. > However, this is a mess which *will* cause trouble later on, so a work > item (perhaps for 0.3 milestone) should be to sort out this (and a number > of other messy bits in the expression system). I tentatively marked a couple of bugs related to expression compilation and evaluation as 0.3 milestones. Though I haven't really thought much about what happens after 0.2, and probably won't until all the 0.2 stuff is done. -- From hategan at mcs.anl.gov Wed Apr 25 09:20:31 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 09:20:31 -0500 (CDT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177448973.16453.1.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Ben Clifford wrote: > > > On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > It's been hacked around for a while now: > > > int[] r = [1:2:1]; > > for i in r { > > } > > Do you mean foreach there? Yes. Should have spelled that out: create an array of integers with the range construct then iterate over that with the iterator for arrays. > > If not, then that doesn't correspond with my view of the history at all - > the VDL.g grammar lost its production rule for for(;;) in r375, and > immediately before that it looked like: > > "for" LPAREN e1=assignStat SEMI e2=expression SEMI e3=assignStat RPAREN > > which is for(a;b;c) syntax. > > -- > > From hategan at mcs.anl.gov Wed Apr 25 09:22:58 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 09:22:58 -0500 (CDT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177448973.16453.1.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Ben Clifford wrote: > > > On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > I would though like to see for i in 1..2 working though. > > that should be relatively straight-forward syntactic sugar, but I don't > particularly like introducing it as a special case for foreach - it should > be a synonym everywhere for [1:2:1] if introduced. But bleugh, [1:2] is > only one character more than 1..2 About that. Why are we using 1:2? It's not exactly intuitive. > > -- > > From benc at hawaga.org.uk Wed Apr 25 09:25:03 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 14:25:03 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177448973.16453.1.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > > that should be relatively straight-forward syntactic sugar, but I don't > > particularly like introducing it as a special case for foreach - it should > > be a synonym everywhere for [1:2:1] if introduced. But bleugh, [1:2] is > > only one character more than 1..2 > > About that. Why are we using 1:2? It's not exactly intuitive. I guess because that's how it got written down to begin with long ago. We have far worse things to worry about, though ;-) -- From hategan at mcs.anl.gov Wed Apr 25 09:48:50 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 09:48:50 -0500 Subject: [Swift-devel] Nasty multiparameter function hack. In-Reply-To: References: Message-ID: <1177512530.4752.5.camel@blabla.mcs.anl.gov> On Wed, 2007-04-25 at 08:32 +0000, Ben Clifford wrote: > However, where the parameter is a variable: > > @f(x) > > this turned into a Karajan function call with two parameters, a 'variable' > and a 'path'. In the above case, the variable is x and the path is empty. > > In the case of @f(x.y) then the variable would be x and the path would be > y. That's kinda wrong: @f(x) should be f(getfield(x, path="")) which can be reduced to f(x) and @f(x.y) should be f(getfield(x, path="y")). Consequently: @strcat("x", x) -> strcat("x", x) @strcat("x", x.y) -> strcat("x", getfield(x, path="y")) > > This leads to inconsistencies, such as : > > x = 3 > y = @f(x) > > resulting in a very different karajan function invocation to > > y = @f(3) > > This rather awkward arrangement is used by @filename because it > essentially wants pass-by-reference rather than pass-by-value semantics in > order to extract the filename of its passed parameter. > > I wanted to avoid indulging in massive expression evaluator hackery, so I > put in the following behaviour for multi-parameter functions: > > For a function f with parameters x1, x2, x3, etc which may be constants or > expressions or other function calls or variables (note that all three are > treated differently), then: > > if x_n is a constant, expression or function call, we translate that > into a single karajan-layer parameter. > > if x_n is a variable reference, then we translate that into two karajan > parameters, first the path, and second the variable. > > This magically works for @strcat when used with top level variables, > because path is empty, so @strcat("hi",x) translates into a concatenation > of *three* things: "hi", the path "", and the contents of variable x, and > as "" is a zero wrt @strcat then its the same as "hi" and the value of x. > > However, this approach will fail in at least the following circumstances > (I think) > > i) @strcat("hi",x.y) (because now path will be non-null) > > ii) if we ever have a function that cares about the ordering of its > parameters. > > I don't particularly propose resolving this for the 0.2 timeframe. > However, this is a mess which *will* cause trouble later on, so a work > item (perhaps for 0.3 milestone) should be to sort out this (and a number > of other messy bits in the expression system). > From benc at hawaga.org.uk Wed Apr 25 09:53:01 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 14:53:01 +0000 (GMT) Subject: [Swift-devel] Nasty multiparameter function hack. In-Reply-To: <1177512530.4752.5.camel@blabla.mcs.anl.gov> References: <1177512530.4752.5.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Mihael Hategan wrote: > On Wed, 2007-04-25 at 08:32 +0000, Ben Clifford wrote: > > However, where the parameter is a variable: > > > > @f(x) > > > > this turned into a Karajan function call with two parameters, a 'variable' > > and a 'path'. In the above case, the variable is x and the path is empty. > > > > In the case of @f(x.y) then the variable would be x and the path would be > > y. > > That's kinda wrong: > @f(x) should be f(getfield(x, path="")) which can be reduced to f(x) and > @f(x.y) should be f(getfield(x, path="y")). > > Consequently: > @strcat("x", x) -> strcat("x", x) > @strcat("x", x.y) -> strcat("x", getfield(x, path="y")) Does getfield still pass through enough info for something like @filename(x.y) to work? I haven't really done anything with getfield ever. -- From hategan at mcs.anl.gov Wed Apr 25 10:04:05 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 10:04:05 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> Message-ID: <1177513446.4752.15.camel@blabla.mcs.anl.gov> That with the assumption that k[i-1] would work. It seems that k[i] = k[i-1] + i gets translated into this: assign(dpath="[i]", dvar=k, svar = sum(getFieldValue(var=k, path=concat("[]")), getFieldValue(i)) ) Seems like k[i-1] is translated into the wrong thing there (path=concat("[]")). This comes from the fact that paths like [i] are passed directly instead of the value being extracted. Yong, do you remember why we did this? On Wed, 2007-04-25 at 07:35 +0000, Ben Clifford wrote: > > On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > Because the jobs are independent. You are creating iteratiion-local > > variables between which there is no dependency. > > > > What you want is something like this: > > file[] lattice <...>; > > lattice[0] = initialLattice; > > for i in 1...2 { > > lattice[i] = app(lattice[i-1]); > > } > > I just tried this: > > int r[] = [1:2:1]; > int k[]; > > k[0]=100; > > foreach i in r { > k[i] = k[i-1] + i; > } > > and I get this: > > $ swift for.swift > Swift V 0.0405 > RunID: 2hp99svgoyjd2 > Execution failed: > String index out of range: 0 > > > :-( > From hategan at mcs.anl.gov Wed Apr 25 10:06:08 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 10:06:08 -0500 Subject: [Swift-devel] Nasty multiparameter function hack. In-Reply-To: References: <1177512530.4752.5.camel@blabla.mcs.anl.gov> Message-ID: <1177513568.4752.17.camel@blabla.mcs.anl.gov> On Wed, 2007-04-25 at 14:53 +0000, Ben Clifford wrote: > > On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > On Wed, 2007-04-25 at 08:32 +0000, Ben Clifford wrote: > > > However, where the parameter is a variable: > > > > > > @f(x) > > > > > > this turned into a Karajan function call with two parameters, a 'variable' > > > and a 'path'. In the above case, the variable is x and the path is empty. > > > > > > In the case of @f(x.y) then the variable would be x and the path would be > > > y. > > > > That's kinda wrong: > > @f(x) should be f(getfield(x, path="")) which can be reduced to f(x) and > > @f(x.y) should be f(getfield(x, path="y")). > > > > Consequently: > > @strcat("x", x) -> strcat("x", x) > > @strcat("x", x.y) -> strcat("x", getfield(x, path="y")) > > Does getfield still pass through enough info for something like > @filename(x.y) to work? it should be the same as tmp = x.y; @filename(tmp); So yes. In contrast to getFieldValue, it does not unwrap things. > > I haven't really done anything with getfield ever. > From hategan at mcs.anl.gov Wed Apr 25 10:08:22 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 10:08:22 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177513446.4752.15.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177513446.4752.15.camel@blabla.mcs.anl.gov> Message-ID: <1177513703.4752.20.camel@blabla.mcs.anl.gov> On Wed, 2007-04-25 at 10:04 -0500, Mihael Hategan wrote: > That with the assumption that k[i-1] would work. > It seems that k[i] = k[i-1] + i gets translated into this: > assign(dpath="[i]", dvar=k, > svar = sum(getFieldValue(var=k, path=concat("[]")), getFieldValue(i)) > ) > > Seems like k[i-1] is translated into the wrong thing there > (path=concat("[]")). It could, instead, be translated into: concat("[", subtraction(i, 1), "]"). My guess is that would work. > This comes from the fact that paths like [i] are passed directly instead > of the value being extracted. > > Yong, do you remember why we did this? > > > On Wed, 2007-04-25 at 07:35 +0000, Ben Clifford wrote: > > > > On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > > > Because the jobs are independent. You are creating iteratiion-local > > > variables between which there is no dependency. > > > > > > What you want is something like this: > > > file[] lattice <...>; > > > lattice[0] = initialLattice; > > > for i in 1...2 { > > > lattice[i] = app(lattice[i-1]); > > > } > > > > I just tried this: > > > > int r[] = [1:2:1]; > > int k[]; > > > > k[0]=100; > > > > foreach i in r { > > k[i] = k[i-1] + i; > > } > > > > and I get this: > > > > $ swift for.swift > > Swift V 0.0405 > > RunID: 2hp99svgoyjd2 > > Execution failed: > > String index out of range: 0 > > > > > > :-( > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From yongzh at cs.uchicago.edu Wed Apr 25 11:07:27 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Wed, 25 Apr 2007 11:07:27 -0500 (CDT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177513446.4752.15.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177513446.4752.15.camel@blabla.mcs.anl.gov> Message-ID: There was no particular reason, I just don't remember whether we allowed expression as array index. If so, that should be translated into path=concat("[", subtract(i,1), "]") Yong. On Wed, 25 Apr 2007, Mihael Hategan wrote: > That with the assumption that k[i-1] would work. > It seems that k[i] = k[i-1] + i gets translated into this: > assign(dpath="[i]", dvar=k, > svar = sum(getFieldValue(var=k, path=concat("[]")), getFieldValue(i)) > ) > > Seems like k[i-1] is translated into the wrong thing there > (path=concat("[]")). > This comes from the fact that paths like [i] are passed directly instead > of the value being extracted. > > Yong, do you remember why we did this? > > > On Wed, 2007-04-25 at 07:35 +0000, Ben Clifford wrote: > > > > On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > > > Because the jobs are independent. You are creating iteratiion-local > > > variables between which there is no dependency. > > > > > > What you want is something like this: > > > file[] lattice <...>; > > > lattice[0] = initialLattice; > > > for i in 1...2 { > > > lattice[i] = app(lattice[i-1]); > > > } > > > > I just tried this: > > > > int r[] = [1:2:1]; > > int k[]; > > > > k[0]=100; > > > > foreach i in r { > > k[i] = k[i-1] + i; > > } > > > > and I get this: > > > > $ swift for.swift > > Swift V 0.0405 > > RunID: 2hp99svgoyjd2 > > Execution failed: > > String index out of range: 0 > > > > > > :-( > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From yongzh at cs.uchicago.edu Wed Apr 25 11:13:11 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Wed, 25 Apr 2007 11:13:11 -0500 (CDT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177448973.16453.1.camel@blabla.mcs.anl.gov> Message-ID: without step 1..2 is fine, but the syntactic sugar does not give you type info. So you need to figure out whether it is integer or float or sth else. with step info, what do you make it, 1..2..1? 1..2:1? so I was using the matlab style. Maybe we can simplify it without the [] around. So we have int i[] = 1:2:1; Yong. On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > On Wed, 25 Apr 2007, Ben Clifford wrote: > > > > > > > On Tue, 24 Apr 2007, Mihael Hategan wrote: > > > > > I would though like to see for i in 1..2 working though. > > > > that should be relatively straight-forward syntactic sugar, but I don't > > particularly like introducing it as a special case for foreach - it should > > be a synonym everywhere for [1:2:1] if introduced. But bleugh, [1:2] is > > only one character more than 1..2 > > About that. Why are we using 1:2? It's not exactly intuitive. > > > > > -- > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From nefedova at mcs.anl.gov Wed Apr 25 11:26:57 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Wed, 25 Apr 2007 11:26:57 -0500 Subject: [Swift-devel] random seed generator? Message-ID: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> Hi, I know I've raised this question before and I had received several suggestions, but I am wondering what is the consensus on generating the random seed in swift? I needed it for MolDyn workflow (had to do some hacks for them), and I need it for the lqcd workflow too. For lqcd I need to change the random seed inside the input file at every loop step. What is the best way to proceed? I see a couple of options: 1. have an external program generate the random seed and then call it from swift. 2. Have swift somehow generate the random seed I am curious if option 2 is possible ? Thanks, Nika From tiberius at ci.uchicago.edu Wed Apr 25 11:36:38 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Wed, 25 Apr 2007 11:36:38 -0500 Subject: [Swift-devel] random seed generator? In-Reply-To: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> Message-ID: I'd go with the external source. In the end, to achieve high quality randomness, swift would probably use the /dev/random internally anyway, so it's just the issue of where to use the random generator in the code. Since swift is not intended to contain non-workflow libraries, i think we should not expect random() to be part of it Tibi On 4/25/07, Veronika V. Nefedova wrote: > Hi, > > I know I've raised this question before and I had received several > suggestions, but I am wondering what is the consensus on generating the > random seed in swift? I needed it for MolDyn workflow (had to do some hacks > for them), and I need it for the lqcd workflow too. > For lqcd I need to change the random seed inside the input file at every > loop step. What is the best way to proceed? I see a couple of options: > > 1. have an external program generate the random seed and then call it from > swift. > 2. Have swift somehow generate the random seed > > I am curious if option 2 is possible ? > > Thanks, > > Nika > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From benc at hawaga.org.uk Wed Apr 25 11:25:31 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 16:25:31 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177513446.4752.15.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Yong Zhao wrote: > There was no particular reason, I just don't remember whether we allowed > expression as array index. If so, that should be translated into > > path=concat("[", subtract(i,1), "]") Not sure if it was intended in the past, but it looks like it would be useful, so I'll play round with this a bit. -- From benc at hawaga.org.uk Wed Apr 25 11:43:45 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 16:43:45 +0000 (GMT) Subject: [Swift-devel] random seed generator? In-Reply-To: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Veronika V. Nefedova wrote: > I know I've raised this question before and I had received several > suggestions, but I am wondering what is the consensus on generating the random > seed in swift? I needed it for MolDyn workflow (had to do some hacks for > them), and I need it for the lqcd workflow too. > For lqcd I need to change the random seed inside the input file at every loop > step. What is the best way to proceed? I see a couple of options: > > 1. have an external program generate the random seed and then call it from > swift. > 2. Have swift somehow generate the random seed > > I am curious if option 2 is possible ? >From a practical perspective, we could have a @random function which generates a random number - I'll make one. >From a theoretical perspective, it changes the notion of reproducibility to be more deliberately non-deterministic; though this isn't the only source of such non-determinism. For example, different software versions will produce different outputs (for example, gif->jpeg encoders) that are 'the same' under a weaker definition of 'sameness' that means 'they look the same to a person', and thus I guess your outputs would be 'the same' under a statistical definition. -- From hategan at mcs.anl.gov Wed Apr 25 11:42:18 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 11:42:18 -0500 Subject: [Swift-devel] random seed generator? In-Reply-To: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> Message-ID: <1177519338.8988.0.camel@blabla.mcs.anl.gov> I was trying: string a; a = random(); But for some reason this translates into random(a). On Wed, 2007-04-25 at 11:26 -0500, Veronika V. Nefedova wrote: > Hi, > > I know I've raised this question before and I had received several > suggestions, but I am wondering what is the consensus on generating the > random seed in swift? I needed it for MolDyn workflow (had to do some hacks > for them), and I need it for the lqcd workflow too. > For lqcd I need to change the random seed inside the input file at every > loop step. What is the best way to proceed? I see a couple of options: > > 1. have an external program generate the random seed and then call it from > swift. > 2. Have swift somehow generate the random seed > > I am curious if option 2 is possible ? > > Thanks, > > Nika > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From benc at hawaga.org.uk Wed Apr 25 11:45:35 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 16:45:35 +0000 (GMT) Subject: [Swift-devel] random seed generator? In-Reply-To: <1177519338.8988.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> <1177519338.8988.0.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Mihael Hategan wrote: > I was trying: > string a; > a = random(); > > But for some reason this translates into random(a). @random not random perhaps. -- From benc at hawaga.org.uk Wed Apr 25 11:47:21 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 16:47:21 +0000 (GMT) Subject: [Swift-devel] random seed generator? In-Reply-To: References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> <1177519338.8988.0.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Ben Clifford wrote: > > I was trying: > > string a; > > a = random(); > > > > But for some reason this translates into random(a). > > @random > > not > > random > > perhaps. Though you'll hit a namespace problem beause it will compile to vdl:random. In which case, there's a ./src/org/griphyn/vdl/karajan/lib/Misc.java which I put in for strcat which could be changed. -- From hategan at mcs.anl.gov Wed Apr 25 11:47:43 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 11:47:43 -0500 Subject: [Swift-devel] random seed generator? In-Reply-To: References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> <1177519338.8988.0.camel@blabla.mcs.anl.gov> Message-ID: <1177519663.9741.1.camel@blabla.mcs.anl.gov> On Wed, 2007-04-25 at 16:45 +0000, Ben Clifford wrote: > > On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > I was trying: > > string a; > > a = random(); > > > > But for some reason this translates into random(a). > > @random that yields "unexpected token". And it would translate to vdl:random(). I wanted to avoid having an explicit namespace, so that it would pick math:random(). > > not > > random > > perhaps. > From benc at hawaga.org.uk Wed Apr 25 11:53:59 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 16:53:59 +0000 (GMT) Subject: [Swift-devel] random seed generator? In-Reply-To: <1177519663.9741.1.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> <1177519338.8988.0.camel@blabla.mcs.anl.gov> <1177519663.9741.1.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Mihael Hategan wrote: > > On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > > > I was trying: > > > string a; > > > a = random(); > > > > > > But for some reason this translates into random(a). > > > > @random > > that yields "unexpected token" huh. It seems to not like zero-parameter functions now. I thought it did before. Maybe I broke it. > And it would translate to vdl:random(). I wanted to avoid having an > explicit namespace, so that it would pick math:random(). I don't think you can do that any way at the moment, and I'm not terribly inclined to have it project all of karajan's function space into SwiftScript space by default. -- From hategan at mcs.anl.gov Wed Apr 25 11:56:49 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 11:56:49 -0500 Subject: [Swift-devel] random seed generator? In-Reply-To: References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> <1177519338.8988.0.camel@blabla.mcs.anl.gov> <1177519663.9741.1.camel@blabla.mcs.anl.gov> Message-ID: <1177520210.9741.7.camel@blabla.mcs.anl.gov> On Wed, 2007-04-25 at 16:53 +0000, Ben Clifford wrote: > > On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > > On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > > > > > I was trying: > > > > string a; > > > > a = random(); > > > > > > > > But for some reason this translates into random(a). > > > > > > @random > > > > that yields "unexpected token" > > huh. It seems to not like zero-parameter functions now. I thought it did > before. Maybe I broke it. Hold your horses. I haven't done an update in a while. > > > And it would translate to vdl:random(). I wanted to avoid having an > > explicit namespace, so that it would pick math:random(). > > I don't think you can do that any way at the moment, and I'm not terribly > inclined to have it project all of karajan's function space into > SwiftScript space by default. It may be useful to have a way of explicitly doing that though. > From benc at hawaga.org.uk Wed Apr 25 12:00:17 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 17:00:17 +0000 (GMT) Subject: [Swift-devel] random seed generator? In-Reply-To: <1177520210.9741.7.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> <1177519338.8988.0.camel@blabla.mcs.anl.gov> <1177519663.9741.1.camel@blabla.mcs.anl.gov> <1177520210.9741.7.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Mihael Hategan wrote: > > huh. It seems to not like zero-parameter functions now. I thought it did > > before. Maybe I broke it. > > Hold your horses. I haven't done an update in a while. it broke for me too. I see why. -- From nefedova at mcs.anl.gov Wed Apr 25 12:55:54 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Wed, 25 Apr 2007 12:55:54 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177447633.15483.3.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070425124932.05814ec0@mail.mcs.anl.gov> I tried what Mihael has suggested, but I am not sure that i got all the syntax correct since it doesn't compile (complete dtm file is in /home/nefedova/LQCD/fermi_bench/lqcd-genU.dtm on terminable) file original[]; original[0]="test.in"; file lattice[]; lattice[0]="lattice.1"; foreach i in range { file lattice[i] ; lattice[i] = lqcd_exec(original[i-1],lattice[i-1]); file original[i] <"test.in">; original[i]=mode_test_in(original[i-1],i); } When I try to compile it, it complains about this line: file lattice[i] ; [nefedova at terminable fermi_bench]$ swift lqcd-genU.dtm Could not compile SwiftScript source: line 22:13: unexpected token: [ [nefedova at terminable fermi_bench]$ Please let me know what I did wrong here, Thanks, Nika At 03:47 PM 4/24/2007, Mihael Hategan wrote: >Because the jobs are independent. You are creating iteratiion-local >variables between which there is no dependency. > >What you want is something like this: >file[] lattice <...>; >lattice[0] = initialLattice; >for i in 1...2 { > lattice[i] = app(lattice[i-1]); >} > >On Tue, 2007-04-24 at 15:37 -0500, Veronika V. Nefedova wrote: > > Fortunately, Ti was able to fix my env on terminable (some faulty softenv > > key thats not working in tcsh) so I was able to try the run there. > > The compilation was fine, no problems. Then it submitted the workflow. > > > > The main problem: it submitted 2 jobs at once (foreach loop is for i=1,2). > > And since the jobs were supposed to run one after another - they all > failed > > (application failure) > > > > Nika > > > > > > > > At 03:17 PM 4/24/2007, Ben Clifford wrote: > > > > >GCJ! that isn't normal java. is that what you normally use? it used to be > > >lame, though i have no idea now. also evitable is unsupported. try > > >terminable. > > > > > >On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > You right! here is the output: > > > > > > > > [nefedova at evitable fermi_bench]$ swift -debug lqcd-genU.dtm > > > > lqcd-genU.dtm: source file is new. Recompiling. > > > > Detailed exception: > > > > java.lang.RuntimeException: Failed to convert .xml to .kml for > > > lqcd-genU.dtm > > > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > > > Source) > > > > at org.griphyn.vdl.karajan.Loader.main(java.lang.String[]) (Unknown > > > Source) > > > > at gnu.java.lang.MainThread.call_main() (/usr/lib/libgcj.so.6.0.0) > > > > at gnu.java.lang.MainThread.run() (/usr/lib/libgcj.so.6.0.0) > > > > Caused by: java.lang.NullPointerException > > > > at > > > org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(java.io.File, > > > > org.apache.xmlbeans.SchemaType, org.apache.xmlbeans.XmlOptions) > (Unknown > > > > Source) > > > > at org.griphyn.vdl.model.ProgramDocument$Factory.parse(java.io.File) > > > > (Unknown Source) > > > > at org.griphyn.vdl.engine.Karajan.main(java.lang.String[]) (Unknown > > > Source) > > > > at org.griphyn.vdl.karajan.Loader.compile(java.lang.String) (Unknown > > > > Source) > > > > ...3 more > > > > > > > > Could not start execution. > > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > Nika > > > > > > > > At 03:11 PM 4/24/2007, Ben Clifford wrote: > > > > > > > > > run with -debug and you should get a much longer output. > > > > > > > > > > On Tue, 24 Apr 2007, Veronika V. Nefedova wrote: > > > > > > > > > > > I did some modifications to the dtm file, and now i get something > > > > > different: > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > Could not start execution. > > > > > > Failed to convert .xml to .kml for lqcd-genU.dtm > > > > > > [nefedova at evitable fermi_bench]$ pwd > > > > > > /autonfs/home/nefedova/LQCD/fermi_bench > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > Any idea what the problem could be? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Nika > > > > > > > > > > > > At 01:39 PM 4/24/2007, Mihael Hategan wrote: > > > > > > > On Tue, 2007-04-24 at 13:32 -0500, Mihael Hategan wrote: > > > > > > > > Posting the kml file would be useful. > > > > > > > > > > > > > > Riiight. The nonexistent kml file. Nevermind me. > > > > > > > > > > > > > > > > > > > > > > > Mihael > > > > > > > > > > > > > > > > On Tue, 2007-04-24 at 13:23 -0500, Veronika V. Nefedova wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I am wondering why this is not working? > > > > > > > > > > > > > > > > > > foreach i in range { > > > > > > > > > file in > > > > > > > > file out > > > > > > > > out=lqcd_exec(original,in); > > > > > > > > > } > > > > > > > > > > > > > > > > > > Swift complains about file declarations (mappers) inside > foreach > > > > > > > statement: > > > > > > > > > > > > > > > > > > [nefedova at evitable fermi_bench]$ swift lqcd-genU.dtm > > > > > > > > > Could not compile SwiftScript source: line 11:1: unexpected > > > token: > > > > > file > > > > > > > > > [nefedova at evitable fermi_bench]$ > > > > > > > > > > > > > > > > > > Please let me know how to make it working (-; > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > Nika > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > Swift-devel mailing list > > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > Swift-devel mailing list > > > > > > > > Swift-devel at ci.uchicago.edu > > > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Swift-devel mailing list > > > > > > Swift-devel at ci.uchicago.edu > > > > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > From benc at hawaga.org.uk Wed Apr 25 13:13:55 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 25 Apr 2007 18:13:55 +0000 (GMT) Subject: [Swift-devel] random seed generator? In-Reply-To: <1177519663.9741.1.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> <1177519338.8988.0.camel@blabla.mcs.anl.gov> <1177519663.9741.1.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Mihael Hategan wrote: > On Wed, 2007-04-25 at 16:45 +0000, Ben Clifford wrote: > > > > On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > > > I was trying: > > > string a; > > > a = random(); > > > > > > But for some reason this translates into random(a). > > > > @random > > that yields "unexpected token". should be fixed in r656 -- From wilde at mcs.anl.gov Wed Apr 25 13:44:11 2007 From: wilde at mcs.anl.gov (Mike Wilde) Date: Wed, 25 Apr 2007 13:44:11 -0500 Subject: [Swift-devel] Re: [Dsl-seminar] REMINDER: discussion on dCache In-Reply-To: <462F9FBF.70501@cs.uchicago.edu> References: <462F9FBF.70501@cs.uchicago.edu> Message-ID: <462FA17B.6060409@mcs.anl.gov> I'm very interested in understanding dCache and its relationship to SRM and GridFTP better, as this is a critical topic for using OSG and running workflows there. And for doing space management within workflows in general. Rob, Marco and Suchandra, all cc'ed here, might be able to point to the best docs. And there is a growing body of info on the OSG Storage Management wiki pages. It would be great for the Swift team to stay in the loop on this. - Mike Ioan Raicu wrote, On 4/25/2007 1:36 PM: > Hi everyone, > We will have our weekly seminar tomorrow at 4:30PM in RI405. Tomorrow's > topic of discussion is dCache (http://www.dcache.org/index.shtml). I > tried to find some detailed papers on dCache's performance and > scalability, but could not. > > I did find general and short papers about the architecture and features > of dCache: > > * http://www.dcache.org/manuals/dcache-whitepaper-light.pdf > * http://www.dcache.org/manuals/dcache-dresden-sep-2006.pdf > * http://www.dcache.org/manuals/hepix-2006-paper.pdf > > We can discuss these, and if it seems interesting for our group, keep > looking for other more technical papers. If anyone runs across any > technical papers on dCache and perhaps how it compares to GFarm and GFS, > please forward them on to the list. > > See you tomorrow, > Ioan > -- Mike Wilde Computation Institute, University of Chicago Math & Computer Science Division Argonne National Laboratory Argonne, IL 60439 USA tel 630-252-7497 fax 630-252-1997 From nefedova at mcs.anl.gov Wed Apr 25 13:44:42 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Wed, 25 Apr 2007 13:44:42 -0500 Subject: [Swift-devel] app names with "-" Message-ID: <6.0.0.22.2.20070425133830.05810d20@mail.mcs.anl.gov> I did a typo in the name of the procedure: I typed "test-in" instead of "test_in". And started to get a lot of syntax errors. Apparently the "-" is not allowed in swift names? I did a quick test, and it [proved that swift doesn't like the "-": type file {} (file f) ec-ho (string s) { app { echo s stdout=@filename(f); } } file f <"testoutput">; f=ec-ho("BLA"); And this wouldn't compile while it would compile just fine when you do s/ec-ho/echo/g This is probably not right. Nika From hategan at mcs.anl.gov Wed Apr 25 13:47:42 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 25 Apr 2007 13:47:42 -0500 Subject: [Swift-devel] app names with "-" In-Reply-To: <6.0.0.22.2.20070425133830.05810d20@mail.mcs.anl.gov> References: <6.0.0.22.2.20070425133830.05810d20@mail.mcs.anl.gov> Message-ID: <1177526863.14517.0.camel@blabla.mcs.anl.gov> On Wed, 2007-04-25 at 13:44 -0500, Veronika V. Nefedova wrote: > I did a typo in the name of the procedure: I typed "test-in" instead of > "test_in". And started to get a lot of syntax errors. Apparently the "-" is > not allowed in swift names? > I did a quick test, and it [proved that swift doesn't like the "-": > > type file {} > (file f) ec-ho (string s) { > app { > echo s stdout=@filename(f); > } > } > file f <"testoutput">; > f=ec-ho("BLA"); > > And this wouldn't compile while it would compile just fine when you do > s/ec-ho/echo/g > This is probably not right. I think it's fairly reasonable to have operators not be valid identifier characters. > > Nika > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From rwg at hep.uchicago.edu Wed Apr 25 14:13:18 2007 From: rwg at hep.uchicago.edu (Rob Gardner) Date: Wed, 25 Apr 2007 14:13:18 -0500 Subject: [Swift-devel] Re: [Dsl-seminar] REMINDER: discussion on dCache In-Reply-To: <462FA17B.6060409@mcs.anl.gov> References: <462F9FBF.70501@cs.uchicago.edu> <462FA17B.6060409@mcs.anl.gov> Message-ID: <46B6A8E4-DBF1-46BD-8944-DEC509C37B97@hep.uchicago.edu> Mike, We have dcache deployed on our tier2 and tier3 centers here at UC - http://uct2-dc3.uchicago.edu:2288/ but we've not yet done detailed scalability studies. So far it has met our needs but we've not yet hit it too hard. Rob On Apr 25, 2007, at 1:44 PM, Mike Wilde wrote: > I'm very interested in understanding dCache and its relationship to > SRM and GridFTP better, as this is a critical topic for using OSG > and running workflows there. And for doing space management within > workflows in general. > > Rob, Marco and Suchandra, all cc'ed here, might be able to point to > the best docs. > > And there is a growing body of info on the OSG Storage Management > wiki pages. > > It would be great for the Swift team to stay in the loop on this. > > - Mike > > > Ioan Raicu wrote, On 4/25/2007 1:36 PM: >> Hi everyone, >> We will have our weekly seminar tomorrow at 4:30PM in RI405. >> Tomorrow's topic of discussion is dCache (http://www.dcache.org/ >> index.shtml). I tried to find some detailed papers on dCache's >> performance and scalability, but could not. >> I did find general and short papers about the architecture and >> features of dCache: >> * http://www.dcache.org/manuals/dcache-whitepaper-light.pdf >> * http://www.dcache.org/manuals/dcache-dresden-sep-2006.pdf >> * http://www.dcache.org/manuals/hepix-2006-paper.pdf >> We can discuss these, and if it seems interesting for our group, >> keep looking for other more technical papers. If anyone runs >> across any technical papers on dCache and perhaps how it compares >> to GFarm and GFS, please forward them on to the list. See you >> tomorrow, >> Ioan > > -- > Mike Wilde > Computation Institute, University of Chicago > Math & Computer Science Division > Argonne National Laboratory > Argonne, IL 60439 USA > tel 630-252-7497 fax 630-252-1997 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2644 bytes Desc: not available URL: From iraicu at cs.uchicago.edu Wed Apr 25 15:17:14 2007 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Wed, 25 Apr 2007 15:17:14 -0500 Subject: [Swift-devel] Re: [Dsl-seminar] REMINDER: discussion on dCache In-Reply-To: <46B6A8E4-DBF1-46BD-8944-DEC509C37B97@hep.uchicago.edu> References: <462F9FBF.70501@cs.uchicago.edu> <462FA17B.6060409@mcs.anl.gov> <46B6A8E4-DBF1-46BD-8944-DEC509C37B97@hep.uchicago.edu> Message-ID: <462FB74A.8000003@cs.uchicago.edu> Hi, Actually, I only heard of dCache when we used the Tier3 cluster for some of the Falkon experiments. After reading up on dCache a bit, it seems like an interesting system, and hence decided to discuss it at the DSL Seminar. I do have some questions. Does dCache run on dedicated machines (similar to the way GPFS does), or does every compute not runs the necessary dCache software to ensure that the compute node handles both the compute and storage needs? I assume each node does both, but the next question is then how do applications do scheduling of application tasks on the compute/storage resources? If done naively, lots of data will likely be moved from one node to another all the time... but if done right, and a cache-ware scheduler was in place, then application tasks could be scheduled on compute nodes that have the most data that is needed. Do you know how this is handled in dCache? Either way, I bet its pretty fast and scalable! I am curious how much work there is left to invest in dCache to have a complete solution that is likely to be the most flexible and offer the best performance improvements given the particular hardware constraints that dCache is deployed on? Thanks, Ioan Rob Gardner wrote: > Mike, > > We have dcache deployed on our tier2 and tier3 centers here at UC - > > http://uct2-dc3.uchicago.edu:2288/ > > but we've not yet done detailed scalability studies. So far it has > met our needs but we've not yet hit it too hard. > > Rob > > > > On Apr 25, 2007, at 1:44 PM, Mike Wilde wrote: > >> I'm very interested in understanding dCache and its relationship to >> SRM and GridFTP better, as this is a critical topic for using OSG and >> running workflows there. And for doing space management within >> workflows in general. >> >> Rob, Marco and Suchandra, all cc'ed here, might be able to point to >> the best docs. >> >> And there is a growing body of info on the OSG Storage Management >> wiki pages. >> >> It would be great for the Swift team to stay in the loop on this. >> >> - Mike >> >> >> Ioan Raicu wrote, On 4/25/2007 1:36 PM: >>> Hi everyone, >>> We will have our weekly seminar tomorrow at 4:30PM in RI405. >>> Tomorrow's topic of discussion is dCache >>> (http://www.dcache.org/index.shtml). I tried to find some detailed >>> papers on dCache's performance and scalability, but could not. >>> I did find general and short papers about the architecture and >>> features of dCache: >>> * http://www.dcache.org/manuals/dcache-whitepaper-light.pdf >>> * http://www.dcache.org/manuals/dcache-dresden-sep-2006.pdf >>> * http://www.dcache.org/manuals/hepix-2006-paper.pdf >>> We can discuss these, and if it seems interesting for our group, >>> keep looking for other more technical papers. If anyone runs across >>> any technical papers on dCache and perhaps how it compares to GFarm >>> and GFS, please forward them on to the list. See you tomorrow, >>> Ioan >> >> -- >> Mike Wilde >> Computation Institute, University of Chicago >> Math & Computer Science Division >> Argonne National Laboratory >> Argonne, IL 60439 USA >> tel 630-252-7497 fax 630-252-1997 > -- ============================================ Ioan Raicu Ph.D. Student ============================================ Distributed Systems Laboratory Computer Science Department University of Chicago 1100 E. 58th Street, Ryerson Hall Chicago, IL 60637 ============================================ Email: iraicu at cs.uchicago.edu Web: http://www.cs.uchicago.edu/~iraicu http://dsl.cs.uchicago.edu/ ============================================ ============================================ From benc at hawaga.org.uk Fri Apr 27 05:08:41 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 27 Apr 2007 10:08:41 +0000 (GMT) Subject: [Swift-devel] random seed generator? In-Reply-To: References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Ben Clifford wrote: > From a practical perspective, we could have a @random function which > generates a random number - I'll make one. For tracking, this is in bugzilla as bug 51. -- From benc at hawaga.org.uk Fri Apr 27 06:40:45 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 27 Apr 2007 11:40:45 +0000 (GMT) Subject: [Swift-devel] Nasty multiparameter function hack. In-Reply-To: <1177512530.4752.5.camel@blabla.mcs.anl.gov> References: <1177512530.4752.5.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Mihael Hategan wrote: > On Wed, 2007-04-25 at 08:32 +0000, Ben Clifford wrote: > > However, where the parameter is a variable: > > > > @f(x) > > > > this turned into a Karajan function call with two parameters, a 'variable' > > and a 'path'. In the above case, the variable is x and the path is empty. > > > > In the case of @f(x.y) then the variable would be x and the path would be > > y. > > That's kinda wrong: > @f(x) should be f(getfield(x, path="")) which can be reduced to f(x) and > @f(x.y) should be f(getfield(x, path="y")). I put this in r657 (without the path="" reduction). -- From tiberius at ci.uchicago.edu Fri Apr 27 06:53:25 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Fri, 27 Apr 2007 06:53:25 -0500 Subject: [Swift-devel] random seed generator? In-Reply-To: References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> Message-ID: I am not sure about starting to throw in random function (such as the random function !) in Swift. In my mind, swift is all about orchestration, and nothing about application libraries. Tibi On 4/27/07, Ben Clifford wrote: > > > On Wed, 25 Apr 2007, Ben Clifford wrote: > > > From a practical perspective, we could have a @random function which > > generates a random number - I'll make one. > > For tracking, this is in bugzilla as bug 51. > > -- > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From hategan at mcs.anl.gov Fri Apr 27 09:15:36 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 27 Apr 2007 09:15:36 -0500 Subject: [Swift-devel] random seed generator? In-Reply-To: References: <6.0.0.22.2.20070425111952.057ca630@mail.mcs.anl.gov> Message-ID: <1177683336.11071.0.camel@blabla.mcs.anl.gov> On Fri, 2007-04-27 at 06:53 -0500, Tiberiu Stef-Praun wrote: > I am not sure about starting to throw in random function (such as the > random function !) in Swift. > In my mind, swift is all about orchestration, and nothing about > application libraries. Right. And orchestration has nothing to do with programming and consequently does not need functions of any kind. > > Tibi > > On 4/27/07, Ben Clifford wrote: > > > > > > On Wed, 25 Apr 2007, Ben Clifford wrote: > > > > > From a practical perspective, we could have a @random function which > > > generates a random number - I'll make one. > > > > For tracking, this is in bugzilla as bug 51. > > > > -- > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > From nefedova at mcs.anl.gov Fri Apr 27 10:18:52 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Fri, 27 Apr 2007 10:18:52 -0500 Subject: [Swift-devel] nightly built 070426 Message-ID: <6.0.0.22.2.20070427101721.06294c80@mail.mcs.anl.gov> I upgrade to the latest nightly build 070426 and it failed to compile my dtm file (that compiles fine wiith 070321, for example. Any idea why? [238] wiggum /sandbox/ydeng/alamines > MolDyn.dtm: source file is new. Recompiling. Detailed exception: java.lang.RuntimeException: Failed to convert .xml to .kml for MolDyn.dtm at org.griphyn.vdl.karajan.Loader.compile(Loader.java:209) at org.griphyn.vdl.karajan.Loader.main(Loader.java:108) Caused by: java.util.NoSuchElementException: no such attribute: nil in template context [call_arg] at org.antlr.stringtemplate.StringTemplate.rawSetAttribute(StringTemplate.java:643) at org.antlr.stringtemplate.StringTemplate.setAttribute(StringTemplate.java:539) at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:650) at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:625) at org.griphyn.vdl.engine.Karajan.actualParameter(Karajan.java:445) at org.griphyn.vdl.engine.Karajan.call(Karajan.java:338) at org.griphyn.vdl.engine.Karajan.statements(Karajan.java:300) at org.griphyn.vdl.engine.Karajan.program(Karajan.java:113) at org.griphyn.vdl.engine.Karajan.main(Karajan.java:67) at org.griphyn.vdl.karajan.Loader.compile(Loader.java:199) ... 1 more Could not start execution. Failed to convert .xml to .kml for MolDyn.dtm [1] Exit 1 swift debug MolDyn.dtm Nika From nefedova at mcs.anl.gov Fri Apr 27 10:50:46 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Fri, 27 Apr 2007 10:50:46 -0500 Subject: [Swift-devel] nightly built 070426 In-Reply-To: <6.0.0.22.2.20070427101721.06294c80@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427101721.06294c80@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070427103259.0629f850@mail.mcs.anl.gov> More info: At 10:18 AM 4/27/2007, Veronika V. Nefedova wrote: >I upgrade to the latest nightly build 070426 and it failed to compile my >dtm file (that compiles fine wiith 070321, for example. Any idea why? > >[238] wiggum /sandbox/ydeng/alamines > MolDyn.dtm: source file is new. >Recompiling. >Detailed exception: >java.lang.RuntimeException: Failed to convert .xml to .kml for MolDyn.dtm > at org.griphyn.vdl.karajan.Loader.compile(Loader.java:209) > at org.griphyn.vdl.karajan.Loader.main(Loader.java:108) >Caused by: java.util.NoSuchElementException: no such attribute: nil in >template context [call_arg] > at > org.antlr.stringtemplate.StringTemplate.rawSetAttribute(StringTemplate.java:643) > at > org.antlr.stringtemplate.StringTemplate.setAttribute(StringTemplate.java:539) > at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:650) > at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:625) > at org.griphyn.vdl.engine.Karajan.actualParameter(Karajan.java:445) > at org.griphyn.vdl.engine.Karajan.call(Karajan.java:338) > at org.griphyn.vdl.engine.Karajan.statements(Karajan.java:300) > at org.griphyn.vdl.engine.Karajan.program(Karajan.java:113) > at org.griphyn.vdl.engine.Karajan.main(Karajan.java:67) > at org.griphyn.vdl.karajan.Loader.compile(Loader.java:199) > ... 1 more >Could not start execution. > Failed to convert .xml to .kml for MolDyn.dtm > >[1] Exit 1 swift debug MolDyn.dtm > > >Nika > >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From nefedova at mcs.anl.gov Fri Apr 27 10:58:40 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Fri, 27 Apr 2007 10:58:40 -0500 Subject: [Swift-devel] nightly built 070426 In-Reply-To: <6.0.0.22.2.20070427103259.0629f850@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427101721.06294c80@mail.mcs.anl.gov> <6.0.0.22.2.20070427103259.0629f850@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070427105738.05d23200@mail.mcs.anl.gov> Sorry to litter the list... I just wanted to add that 070424 compiles my dtm file just fine. So something has happened in the last couple of days... Nika At 10:50 AM 4/27/2007, Veronika V. Nefedova wrote: >More info: > >At 10:18 AM 4/27/2007, Veronika V. Nefedova wrote: >>I upgrade to the latest nightly build 070426 and it failed to compile my >>dtm file (that compiles fine wiith 070321, for example. Any idea why? >> >>[238] wiggum /sandbox/ydeng/alamines > MolDyn.dtm: source file is new. >>Recompiling. >>Detailed exception: >>java.lang.RuntimeException: Failed to convert .xml to .kml for MolDyn.dtm >> at org.griphyn.vdl.karajan.Loader.compile(Loader.java:209) >> at org.griphyn.vdl.karajan.Loader.main(Loader.java:108) >>Caused by: java.util.NoSuchElementException: no such attribute: nil in >>template context [call_arg] >> at >> org.antlr.stringtemplate.StringTemplate.rawSetAttribute(StringTemplate.java:643) >> at >> org.antlr.stringtemplate.StringTemplate.setAttribute(StringTemplate.java:539) >> at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:650) >> at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:625) >> at org.griphyn.vdl.engine.Karajan.actualParameter(Karajan.java:445) >> at org.griphyn.vdl.engine.Karajan.call(Karajan.java:338) >> at org.griphyn.vdl.engine.Karajan.statements(Karajan.java:300) >> at org.griphyn.vdl.engine.Karajan.program(Karajan.java:113) >> at org.griphyn.vdl.engine.Karajan.main(Karajan.java:67) >> at org.griphyn.vdl.karajan.Loader.compile(Loader.java:199) >> ... 1 more >>Could not start execution. >> Failed to convert .xml to .kml for MolDyn.dtm >> >>[1] Exit 1 swift debug MolDyn.dtm >> >> >>Nika >> >>_______________________________________________ >>Swift-devel mailing list >>Swift-devel at ci.uchicago.edu >>http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From benc at hawaga.org.uk Fri Apr 27 11:48:56 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 27 Apr 2007 16:48:56 +0000 (GMT) Subject: [Swift-devel] nightly built 070426 In-Reply-To: <6.0.0.22.2.20070427101721.06294c80@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427101721.06294c80@mail.mcs.anl.gov> Message-ID: On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > I upgrade to the latest nightly build 070426 and it failed to compile my dtm > file (that compiles fine wiith 070321, for example. Any idea why? I've seen that bug - but I thought its basically a change where something that didn't work before now doesn't work in a different way (previously it behaved wrongly at run time; now it compiles wrong). see bug 49 - http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=49 What's the source file? > > [238] wiggum /sandbox/ydeng/alamines > MolDyn.dtm: source file is new. > Recompiling. > Detailed exception: > java.lang.RuntimeException: Failed to convert .xml to .kml for MolDyn.dtm > at org.griphyn.vdl.karajan.Loader.compile(Loader.java:209) > at org.griphyn.vdl.karajan.Loader.main(Loader.java:108) > Caused by: java.util.NoSuchElementException: no such attribute: nil in > template context [call_arg] > at > org.antlr.stringtemplate.StringTemplate.rawSetAttribute(StringTemplate.java:643) > at > org.antlr.stringtemplate.StringTemplate.setAttribute(StringTemplate.java:539) > at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:650) > at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:625) > at org.griphyn.vdl.engine.Karajan.actualParameter(Karajan.java:445) > at org.griphyn.vdl.engine.Karajan.call(Karajan.java:338) > at org.griphyn.vdl.engine.Karajan.statements(Karajan.java:300) > at org.griphyn.vdl.engine.Karajan.program(Karajan.java:113) > at org.griphyn.vdl.engine.Karajan.main(Karajan.java:67) > at org.griphyn.vdl.karajan.Loader.compile(Loader.java:199) > ... 1 more > Could not start execution. > Failed to convert .xml to .kml for MolDyn.dtm > > [1] Exit 1 swift debug MolDyn.dtm > > > Nika > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From benc at hawaga.org.uk Fri Apr 27 11:33:25 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 27 Apr 2007 16:33:25 +0000 (GMT) Subject: [Swift-devel] spam on bug 13 Message-ID: someone just put a bunch of spam attachments on bug 13. the same (or similar) happen on the globus bugzilla the other day. and on the swift trac. I guess maybe open signup needs to go away. (as predicted long ago by the inventor of tinyurlfs...) -- From nefedova at mcs.anl.gov Fri Apr 27 12:03:25 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Fri, 27 Apr 2007 12:03:25 -0500 Subject: [Swift-devel] nightly built 070426 In-Reply-To: References: <6.0.0.22.2.20070427101721.06294c80@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070427120148.05d24c90@mail.mcs.anl.gov> Thats my dtm MolDyn.dtm file (it in ~nefedova/public_html on CI system)... I am using a 2 days old nightly built now (0424) and it seems to work fine... Nika At 11:48 AM 4/27/2007, Ben Clifford wrote: >On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > > > I upgrade to the latest nightly build 070426 and it failed to compile > my dtm > > file (that compiles fine wiith 070321, for example. Any idea why? > >I've seen that bug - but I thought its basically a change where something >that didn't work before now doesn't work in a different way (previously it >behaved wrongly at run time; now it compiles wrong). > >see bug 49 - http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=49 > >What's the source file? > > > > > > > [238] wiggum /sandbox/ydeng/alamines > MolDyn.dtm: source file is new. > > Recompiling. > > Detailed exception: > > java.lang.RuntimeException: Failed to convert .xml to .kml for MolDyn.dtm > > at org.griphyn.vdl.karajan.Loader.compile(Loader.java:209) > > at org.griphyn.vdl.karajan.Loader.main(Loader.java:108) > > Caused by: java.util.NoSuchElementException: no such attribute: nil in > > template context [call_arg] > > at > > > org.antlr.stringtemplate.StringTemplate.rawSetAttribute(StringTemplate.java:643) > > at > > > org.antlr.stringtemplate.StringTemplate.setAttribute(StringTemplate.java:539) > > at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:650) > > at org.griphyn.vdl.engine.Karajan.setExprOrValue(Karajan.java:625) > > at org.griphyn.vdl.engine.Karajan.actualParameter(Karajan.java:445) > > at org.griphyn.vdl.engine.Karajan.call(Karajan.java:338) > > at org.griphyn.vdl.engine.Karajan.statements(Karajan.java:300) > > at org.griphyn.vdl.engine.Karajan.program(Karajan.java:113) > > at org.griphyn.vdl.engine.Karajan.main(Karajan.java:67) > > at org.griphyn.vdl.karajan.Loader.compile(Loader.java:199) > > ... 1 more > > Could not start execution. > > Failed to convert .xml to .kml for MolDyn.dtm > > > > [1] Exit 1 swift debug MolDyn.dtm > > > > > > Nika > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > From hategan at mcs.anl.gov Fri Apr 27 12:22:33 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 27 Apr 2007 12:22:33 -0500 Subject: [Swift-devel] bugzilla Message-ID: <1177694553.17487.1.camel@blabla.mcs.anl.gov> How does one ban a user from bugzilla? http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=2 From nefedova at mcs.anl.gov Fri Apr 27 13:23:25 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Fri, 27 Apr 2007 13:23:25 -0500 Subject: [Swift-devel] swift in SVN is broken Message-ID: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> Hi, I tried to use swift from svn trunk and the build has failed with these errors. Please fix! Its a blocker for me... Thanks! Nika compile: [echo] [vdsk]: COMPILE [mkdir] Created dir: /sandbox/nefedova/SWIFT/cog/modules/vdsk/build [javac] Compiling 165 source files to /sandbox/nefedova/SWIFT/cog/modules/vdsk/build [javac] /sandbox/nefedova/SWIFT/cog/modules/vdsk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java:35: cannot resolve symbol [javac] symbol : constructor IllegalArgumentException (java.lang.String,org.griphyn.vdl.mapping.InvalidMapperException) [javac] location: class java.lang.IllegalArgumentException [javac] throw new IllegalArgumentException(e.getMessage(), e); [javac] ^ [javac] /sandbox/nefedova/SWIFT/cog/modules/vdsk/src/org/griphyn/vdl/mapping/RootDataNode.java:38: cannot resolve symbol [javac] symbol : constructor IllegalArgumentException (java.lang.String,org.griphyn.vdl.mapping.InvalidMapperException) [javac] location: class java.lang.IllegalArgumentException [javac] throw new IllegalArgumentException(e.getMessage(),e); [javac] ^ [javac] 2 errors BUILD FAILED /sandbox/nefedova/SWIFT/cog/modules/vdsk/build.xml:69: The following error occurred while executing this line: /sandbox/nefedova/SWIFT/cog/mbuild.xml:463: The following error occurred while executing this line: /sandbox/nefedova/SWIFT/cog/mbuild.xml:227: Compile failed; see the compiler error output for details. From hategan at mcs.anl.gov Fri Apr 27 13:25:11 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 27 Apr 2007 13:25:11 -0500 Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> Message-ID: <1177698312.20589.0.camel@blabla.mcs.anl.gov> Note: that constructor has only been added in java 1.5. It's not available in 1.4. On Fri, 2007-04-27 at 13:23 -0500, Veronika V. Nefedova wrote: > Hi, > > I tried to use swift from svn trunk and the build has failed with these > errors. Please fix! Its a blocker for me... > > Thanks! > > Nika > > compile: > [echo] [vdsk]: COMPILE > [mkdir] Created dir: /sandbox/nefedova/SWIFT/cog/modules/vdsk/build > [javac] Compiling 165 source files to > /sandbox/nefedova/SWIFT/cog/modules/vdsk/build > [javac] > /sandbox/nefedova/SWIFT/cog/modules/vdsk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java:35: > cannot resolve symbol > [javac] symbol : constructor IllegalArgumentException > (java.lang.String,org.griphyn.vdl.mapping.InvalidMapperException) > [javac] location: class java.lang.IllegalArgumentException > [javac] throw new > IllegalArgumentException(e.getMessage(), e); > [javac] ^ > [javac] > /sandbox/nefedova/SWIFT/cog/modules/vdsk/src/org/griphyn/vdl/mapping/RootDataNode.java:38: > cannot resolve symbol > [javac] symbol : constructor IllegalArgumentException > (java.lang.String,org.griphyn.vdl.mapping.InvalidMapperException) > [javac] location: class java.lang.IllegalArgumentException > [javac] throw new > IllegalArgumentException(e.getMessage(),e); > [javac] ^ > [javac] 2 errors > > BUILD FAILED > /sandbox/nefedova/SWIFT/cog/modules/vdsk/build.xml:69: The following error > occurred while executing this line: > /sandbox/nefedova/SWIFT/cog/mbuild.xml:463: The following error occurred > while executing this line: > /sandbox/nefedova/SWIFT/cog/mbuild.xml:227: Compile failed; see the > compiler error output for details. > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From tiberius at ci.uchicago.edu Fri Apr 27 13:36:37 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Fri, 27 Apr 2007 13:36:37 -0500 Subject: [Swift-devel] wishlist: string array initialization in a loop Message-ID: ======= string nameArray[]; string channelNo[]= ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", foreach string ch, ci in channelNo{ nameArray[ci]=@strcat(trialType,"-",subjectNo,"channel",ch,"_cwt-avgResults.Rdata"); //print(outputNamesArray[ci]); } string testName=nameArray[3]; print(testName); // returns something like nameArray-6a860c63-62c2-44ce-9fda-6bb43e3c871e-0-0-0-0-0.3 ======= I also tried moving the loop in a procedure, and doing string nameArray[]=loopNamingProcedure(); That did not work either. Any suggestions ? Ben was working on something related, but I still can't do a string array initialization in a loop Tibi -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From nefedova at mcs.anl.gov Fri Apr 27 13:45:00 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Fri, 27 Apr 2007 13:45:00 -0500 Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <1177698312.20589.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> I followed Mihael's advise and was able to compile with java 1.5 Nika At 01:25 PM 4/27/2007, Mihael Hategan wrote: >Note: that constructor has only been added in java 1.5. It's not >available in 1.4. > >On Fri, 2007-04-27 at 13:23 -0500, Veronika V. Nefedova wrote: > > Hi, > > > > I tried to use swift from svn trunk and the build has failed with these > > errors. Please fix! Its a blocker for me... > > > > Thanks! > > > > Nika > > > > compile: > > [echo] [vdsk]: COMPILE > > [mkdir] Created dir: /sandbox/nefedova/SWIFT/cog/modules/vdsk/build > > [javac] Compiling 165 source files to > > /sandbox/nefedova/SWIFT/cog/modules/vdsk/build > > [javac] > > > /sandbox/nefedova/SWIFT/cog/modules/vdsk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java:35: > > > cannot resolve symbol > > [javac] symbol : constructor IllegalArgumentException > > (java.lang.String,org.griphyn.vdl.mapping.InvalidMapperException) > > [javac] location: class java.lang.IllegalArgumentException > > [javac] throw new > > IllegalArgumentException(e.getMessage(), e); > > [javac] ^ > > [javac] > > > /sandbox/nefedova/SWIFT/cog/modules/vdsk/src/org/griphyn/vdl/mapping/RootDataNode.java:38: > > > cannot resolve symbol > > [javac] symbol : constructor IllegalArgumentException > > (java.lang.String,org.griphyn.vdl.mapping.InvalidMapperException) > > [javac] location: class java.lang.IllegalArgumentException > > [javac] throw new > > IllegalArgumentException(e.getMessage(),e); > > [javac] ^ > > [javac] 2 errors > > > > BUILD FAILED > > /sandbox/nefedova/SWIFT/cog/modules/vdsk/build.xml:69: The following error > > occurred while executing this line: > > /sandbox/nefedova/SWIFT/cog/mbuild.xml:463: The following error occurred > > while executing this line: > > /sandbox/nefedova/SWIFT/cog/mbuild.xml:227: Compile failed; see the > > compiler error output for details. > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > From nefedova at mcs.anl.gov Fri Apr 27 13:56:37 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Fri, 27 Apr 2007 13:56:37 -0500 Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> Message-ID: <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> Even though the compilation was successful, the execution was not. Swift failed with Karajan exceptions: RunID: z8onnx8li6qo1 Execution failed: Karajan exception: Ambiguous element: getfield. Possible choices: vdl:getfield java:getfield [1] Exit 2 swift MolDyn.dtm [302] wiggum /sandbox/ydeng/alamines > So it still needs to be fixed. Thanks! Nika At 01:45 PM 4/27/2007, Veronika V. Nefedova wrote: >I followed Mihael's advise and was able to compile with java 1.5 > >Nika > >At 01:25 PM 4/27/2007, Mihael Hategan wrote: >>Note: that constructor has only been added in java 1.5. It's not >>available in 1.4. >> >>On Fri, 2007-04-27 at 13:23 -0500, Veronika V. Nefedova wrote: >> > Hi, >> > >> > I tried to use swift from svn trunk and the build has failed with these >> > errors. Please fix! Its a blocker for me... >> > >> > Thanks! >> > >> > Nika >> > >> > compile: >> > [echo] [vdsk]: COMPILE >> > [mkdir] Created dir: /sandbox/nefedova/SWIFT/cog/modules/vdsk/build >> > [javac] Compiling 165 source files to >> > /sandbox/nefedova/SWIFT/cog/modules/vdsk/build >> > [javac] >> > >> /sandbox/nefedova/SWIFT/cog/modules/vdsk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java:35: >> >> > cannot resolve symbol >> > [javac] symbol : constructor IllegalArgumentException >> > (java.lang.String,org.griphyn.vdl.mapping.InvalidMapperException) >> > [javac] location: class java.lang.IllegalArgumentException >> > [javac] throw new >> > IllegalArgumentException(e.getMessage(), e); >> > [javac] ^ >> > [javac] >> > >> /sandbox/nefedova/SWIFT/cog/modules/vdsk/src/org/griphyn/vdl/mapping/RootDataNode.java:38: >> >> > cannot resolve symbol >> > [javac] symbol : constructor IllegalArgumentException >> > (java.lang.String,org.griphyn.vdl.mapping.InvalidMapperException) >> > [javac] location: class java.lang.IllegalArgumentException >> > [javac] throw new >> > IllegalArgumentException(e.getMessage(),e); >> > [javac] ^ >> > [javac] 2 errors >> > >> > BUILD FAILED >> > /sandbox/nefedova/SWIFT/cog/modules/vdsk/build.xml:69: The following error >> > occurred while executing this line: >> > /sandbox/nefedova/SWIFT/cog/mbuild.xml:463: The following error occurred >> > while executing this line: >> > /sandbox/nefedova/SWIFT/cog/mbuild.xml:227: Compile failed; see the >> > compiler error output for details. >> > >> > _______________________________________________ >> > Swift-devel mailing list >> > Swift-devel at ci.uchicago.edu >> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >> > > > >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From benc at hawaga.org.uk Sat Apr 28 03:16:35 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 28 Apr 2007 08:16:35 +0000 (GMT) Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <1177698312.20589.0.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> Message-ID: On Fri, 27 Apr 2007, Mihael Hategan wrote: > Note: that constructor has only been added in java 1.5. It's not > available in 1.4. oops. I'll fix that - there's still a few places that have no 1.5 available, I think, so keeping 1.4 compatibility is probably good. -- From benc at hawaga.org.uk Sat Apr 28 03:19:24 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 28 Apr 2007 08:19:24 +0000 (GMT) Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> Message-ID: On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > Even though the compilation was successful, the execution was not. Swift > failed with Karajan exceptions: > > RunID: z8onnx8li6qo1 > > Execution failed: > Karajan exception: Ambiguous element: getfield. Possible choices: > vdl:getfield > java:getfield hmm. I don't get such an error on my jdk1.5 - no idea what java:getfield is though - something from the bowels of Karajan, I guess. I'll change the way that getfield is referred to and see if that fixes this for you. -- From benc at hawaga.org.uk Sat Apr 28 03:39:03 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 28 Apr 2007 08:39:03 +0000 (GMT) Subject: [Swift-devel] nightly built 070426 In-Reply-To: <6.0.0.22.2.20070427120148.05d24c90@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427101721.06294c80@mail.mcs.anl.gov> <6.0.0.22.2.20070427120148.05d24c90@mail.mcs.anl.gov> Message-ID: On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > Thats my dtm MolDyn.dtm file (it in ~nefedova/public_html on CI system)... did you do something weird to that file to copy it to public_html? it seems to be wrapped at 90 columns in the middle of tokens, at least when I look at it. The line: ( am1_file_m001) = Pre_antchmbr (@files); should probably be ( am1_file_m001) = Pre_antchmbr (files); without the @ files is an array of type file[], and that is what Pre_antchmbr wants as a parameter. I suspect it was only working for you because bug 49 exists. Our rigourous compile time type checking will one day detect this sort of thing! -- From tiberius at ci.uchicago.edu Sat Apr 28 08:59:13 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Sat, 28 Apr 2007 08:59:13 -0500 Subject: [Swift-devel] swift in SVN is broken In-Reply-To: References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> Message-ID: I missed this, but I'm getting the same error (getfield), and I had to revert my code back to an older version (not the bleeding-edge that Ben has me test for him). Tibi On 4/28/07, Ben Clifford wrote: > > > On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > > > Even though the compilation was successful, the execution was not. Swift > > failed with Karajan exceptions: > > > > RunID: z8onnx8li6qo1 > > > > Execution failed: > > Karajan exception: Ambiguous element: getfield. Possible choices: > > vdl:getfield > > java:getfield > > hmm. I don't get such an error on my jdk1.5 - no idea what java:getfield > is though - something from the bowels of Karajan, I guess. > > I'll change the way that getfield is referred to and see if that fixes > this for you. > > -- > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From tiberius at ci.uchicago.edu Sat Apr 28 09:00:12 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Sat, 28 Apr 2007 09:00:12 -0500 Subject: [Swift-devel] swift in SVN is broken In-Reply-To: References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> Message-ID: I missed this, but I'm getting the same error (getfield), and I had to revert my code back to an older version of Swift (not the bleeding-edge that Ben has me test for him). Tibi > > On 4/28/07, Ben Clifford wrote: > > > > > > On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > > > > > Even though the compilation was successful, the execution was not. Swift > > > failed with Karajan exceptions: > > > > > > RunID: z8onnx8li6qo1 > > > > > > Execution failed: > > > Karajan exception: Ambiguous element: getfield. Possible choices: > > > vdl:getfield > > > java:getfield > > > > hmm. I don't get such an error on my jdk1.5 - no idea what java:getfield > > is though - something from the bowels of Karajan, I guess. > > > > I'll change the way that getfield is referred to and see if that fixes > > this for you. > > > > -- > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > -- > Tiberiu (Tibi) Stef-Praun, PhD > Research Staff, Computation Institute > 5640 S. Ellis Ave, #405 > University of Chicago > http://www-unix.mcs.anl.gov/~tiberius/ > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From hategan at mcs.anl.gov Sat Apr 28 11:51:45 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 28 Apr 2007 11:51:45 -0500 Subject: [Swift-devel] swift in SVN is broken In-Reply-To: References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> Message-ID: <1177779105.14512.1.camel@blabla.mcs.anl.gov> On Sat, 2007-04-28 at 08:19 +0000, Ben Clifford wrote: > > On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > > > Even though the compilation was successful, the execution was not. Swift > > failed with Karajan exceptions: > > > > RunID: z8onnx8li6qo1 > > > > Execution failed: > > Karajan exception: Ambiguous element: getfield. Possible choices: > > vdl:getfield > > java:getfield > > hmm. I don't get such an error on my jdk1.5 - no idea what java:getfield > is though - something from the bowels of Karajan, I guess. It comes from the java library. It gets the value of a field in a class. It's the same problem we were speaking about recently. > > I'll change the way that getfield is referred to and see if that fixes > this for you. > From benc at hawaga.org.uk Sat Apr 28 12:10:58 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 28 Apr 2007 17:10:58 +0000 (GMT) Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <1177779105.14512.1.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> <1177779105.14512.1.camel@blabla.mcs.anl.gov> Message-ID: On Sat, 28 Apr 2007, Mihael Hategan wrote: > It comes from the java library. It gets the value of a field in a class. > It's the same problem we were speaking about recently. Any idea why it isn't happening for me? -- From hategan at mcs.anl.gov Sat Apr 28 12:30:52 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 28 Apr 2007 12:30:52 -0500 Subject: [Swift-devel] swift in SVN is broken In-Reply-To: References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> <1177779105.14512.1.camel@blabla.mcs.anl.gov> Message-ID: <1177781452.15438.4.camel@blabla.mcs.anl.gov> On Sat, 2007-04-28 at 17:10 +0000, Ben Clifford wrote: > > On Sat, 28 Apr 2007, Mihael Hategan wrote: > > > It comes from the java library. It gets the value of a field in a class. > > It's the same problem we were speaking about recently. > > Any idea why it isn't happening for me? Good point. Maybe you had uncommitted changes to vdl.k? > From benc at hawaga.org.uk Sat Apr 28 12:33:39 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 28 Apr 2007 17:33:39 +0000 (GMT) Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> Message-ID: On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > Even though the compilation was successful, the execution was not. Swift > failed with Karajan exceptions: > > RunID: z8onnx8li6qo1 > > Execution failed: > Karajan exception: Ambiguous element: getfield. Possible choices: > vdl:getfield > java:getfield > > > [1] Exit 2 swift MolDyn.dtm > [302] wiggum /sandbox/ydeng/alamines > > > So it still needs to be fixed. > > Thanks! I just added namespaces onto the references to getfield taht I put in the the other day. r660. I haven't tried to replicate the problem,though - it doesn't seem to happen for me, somehow. So I'm unsure if it will fix this problem for you or not. Let me know. -- From benc at hawaga.org.uk Sat Apr 28 12:40:29 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 28 Apr 2007 17:40:29 +0000 (GMT) Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <1177781452.15438.4.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> <1177698312.20589.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070427134431.05384660@mail.mcs.anl.gov> <6.0.0.22.2.20070427135513.05c7c0b0@mail.mcs.anl.gov> <1177779105.14512.1.camel@blabla.mcs.anl.gov> <1177781452.15438.4.camel@blabla.mcs.anl.gov> Message-ID: On Sat, 28 Apr 2007, Mihael Hategan wrote: > On Sat, 2007-04-28 at 17:10 +0000, Ben Clifford wrote: > > > > On Sat, 28 Apr 2007, Mihael Hategan wrote: > > > > > It comes from the java library. It gets the value of a field in a class. > > > It's the same problem we were speaking about recently. > > > > Any idea why it isn't happening for me? > > Good point. Maybe you had uncommitted changes to vdl.k? I regularly take all my patches off (using quilt) and run an svn diff to make sure that I get empty output; so it seems unlikely. If it bothers me on Monday, I'll poke around harder trying to work out why. -- From benc at hawaga.org.uk Sun Apr 29 11:00:54 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Sun, 29 Apr 2007 17:00:54 +0100 (BST) Subject: [Swift-devel] plug points Message-ID: There are a number of places in swift where the 'core' production code needs to interface with other code. The most obvious of these is the interface for non-core mappers. Other places seem to be: * job submission - whilst the rhetoric says 'we will use GRAM', its been the case that we have used at least PBS and DeeF internally for some projects that are not part of the core swift code; so it seems likely that we should write down (and tidy up if necessary) the interface for plugging in different job submission systems. I mostly know how PBS was connected to Swift. I have no idea how DeeF was, and it would perhaps be enlightening if someone would briefly describe how. * site selection - at least one group (the UBA people) have in part neglected using swift because I haven't given them an answer about how to do custom site selection. It seems to me that we should formalise (to a lesser or greater extent) how to plug thing in to the above interfaces - perhaps just a couple of paragraphs on the recommended API, perhaps more. This is in part done for mappers, in the tutorial. At present, additions happen through a range of source level hacks - for example, the mapper tutorial suggests as a final step 'now recompile Swift to make your mapper work'; and there's a lookup table that PBS was added to recently, despite not being a supported submission mechanism. The presence of a couple of pieces of hacked on cruft like this is not really a problem; but its not good in the long term - it makes the codebase as a whole less coherent, and confuses the Swift production code with other (related but not the same) projects. So one of the things I want to do, though not as a high priority, is document the above and make them more coherent/sensible if necessary. -- From hategan at mcs.anl.gov Sun Apr 29 19:26:22 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 29 Apr 2007 19:26:22 -0500 Subject: [Swift-devel] plug points In-Reply-To: References: Message-ID: <1177892782.25695.5.camel@blabla.mcs.anl.gov> On Sun, 2007-04-29 at 17:00 +0100, Ben Clifford wrote: > There are a number of places in swift where the 'core' production code > needs to interface with other code. > > The most obvious of these is the interface for non-core mappers. > > Other places seem to be: > > * job submission - whilst the rhetoric says 'we will use GRAM', its been > the case that we have used at least PBS and DeeF internally for some > projects that are not part of the core swift code; so it seems likely that > we should write down (and tidy up if necessary) the interface for plugging > in different job submission systems. I mostly know how PBS was connected > to Swift. I have no idea how DeeF was, and it would perhaps be > enlightening if someone would briefly describe how. http://wiki.cogkit.org/index.php/Java_CoG_Kit_Abstraction_Guide http://www.cogkit.org/release/current/api/abstraction-common/index.html > > * site selection - at least one group (the UBA people) have in part > neglected using swift because I haven't given them an answer about how to > do custom site selection. Unfortunately this is not that well documented. http://www.cogkit.org/viewcvs/viewcvs.cgi/src/cog/modules/karajan/src/org/globus/cog/karajan/scheduler/Scheduler.java?rev=HEAD&content-type=text/vnd.viewcvs-markup > > It seems to me that we should formalise (to a lesser or greater extent) > how to plug thing in to the above interfaces - perhaps just a couple of > paragraphs on the recommended API, perhaps more. This is in part done for > mappers, in the tutorial. > > At present, additions happen through a range of source level hacks - for > example, the mapper tutorial suggests as a final step 'now recompile Swift > to make your mapper work'; and there's a lookup table that PBS was > added to recently, despite not being a supported submission mechanism. Regarding PBS, it works as pointed above. In principle, given a well defined provider (a provider.properties file) on the classpath, it will be available automatically. > > The presence of a couple of pieces of hacked on cruft like this is not > really a problem; but its not good in the long term - it makes the > codebase as a whole less coherent, and confuses the Swift production code > with other (related but not the same) projects. Right. Fortunately it's one piece that is quite formalized (at least the abstraction stuff). The knowledge about it (conceptual link from Swift to "that thing" is missing). > > So one of the things I want to do, though not as a high priority, is > document the above and make them more coherent/sensible if necessary. > From tiberius at ci.uchicago.edu Sun Apr 29 19:33:15 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Sun, 29 Apr 2007 19:33:15 -0500 Subject: [Swift-devel] swift request Message-ID: file stageOnePrevFiles[]; I needed a parameter as an empty file array, and it seems that swift did not like it -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From benc at hawaga.org.uk Sun Apr 29 19:40:43 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 00:40:43 +0000 (GMT) Subject: [Swift-devel] swift request In-Reply-To: References: Message-ID: On Sun, 29 Apr 2007, Tiberiu Stef-Praun wrote: > file stageOnePrevFiles[]; > > I needed a parameter as an empty file array, and it seems that swift > did not like it What are you trying to do with this? -- From tiberius at ci.uchicago.edu Sun Apr 29 19:45:52 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Sun, 29 Apr 2007 19:45:52 -0500 Subject: [Swift-devel] swift request In-Reply-To: References: Message-ID: I was trying to build a chain, and the first element in the chain did not have input files, so I wanted to pass an empty array. However I solved it by passing a dummy file. Tibi On 4/29/07, Ben Clifford wrote: > > > On Sun, 29 Apr 2007, Tiberiu Stef-Praun wrote: > > > file stageOnePrevFiles[]; > > > > I needed a parameter as an empty file array, and it seems that swift > > did not like it > > What are you trying to do with this? > > -- > > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From benc at hawaga.org.uk Sun Apr 29 19:48:43 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 00:48:43 +0000 (GMT) Subject: [Swift-devel] swift request In-Reply-To: References: Message-ID: paste the rest of your program On Sun, 29 Apr 2007, Tiberiu Stef-Praun wrote: > I was trying to build a chain, and the first element in the chain did > not have input files, so I wanted to pass an empty array. > However I solved it by passing a dummy file. > > Tibi > > On 4/29/07, Ben Clifford wrote: > > > > > > On Sun, 29 Apr 2007, Tiberiu Stef-Praun wrote: > > > > > file stageOnePrevFiles[]; > > > > > > I needed a parameter as an empty file array, and it seems that swift > > > did not like it > > > > What are you trying to do with this? > > > > -- > > > > > > > From benc at hawaga.org.uk Mon Apr 30 05:48:20 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 10:48:20 +0000 (GMT) Subject: [Swift-devel] bugzilla In-Reply-To: <1177694553.17487.1.camel@blabla.mcs.anl.gov> References: <1177694553.17487.1.camel@blabla.mcs.anl.gov> Message-ID: On Fri, 27 Apr 2007, Mihael Hategan wrote: > How does one ban a user from bugzilla? I can, because I have admin rights. However, when I tried that for someone who was spamming one of my bugs, a new user appeared shortly afterwards and carried on. I have a req open with systems about what to do... -- From benc at hawaga.org.uk Mon Apr 30 06:00:03 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 11:00:03 +0000 (GMT) Subject: [Swift-devel] wishlist: string array initialization in a loop In-Reply-To: References: Message-ID: On Fri, 27 Apr 2007, Tiberiu Stef-Praun wrote: > ======= > string nameArray[]; > string channelNo[]= ["1", "2", "3", "4", "5", "6", > "7", "8", "9", "10", "11", "12", "13", "14", > foreach string ch, ci in channelNo{ > > nameArray[ci]=@strcat(trialType,"-",subjectNo,"channel",ch,"_cwt-avgResults.Rdata"); > //print(outputNamesArray[ci]); > } > string testName=nameArray[3]; > print(testName); > // returns something like > nameArray-6a860c63-62c2-44ce-9fda-6bb43e3c871e-0-0-0-0-0.3 > ======= > > I also tried moving the loop in a procedure, and doing > string nameArray[]=loopNamingProcedure(); > > That did not work either. > Any suggestions ? > > Ben was working on something related, but I still can't do a string > array initialization in a loop as far as I can tell, its evaluating this bit: > string testName=nameArray[3]; before it gets round to executing the foreach loop, rather than picking up on the data dependency. hmm. -- From benc at hawaga.org.uk Mon Apr 30 06:21:53 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 11:21:53 +0000 (GMT) Subject: [Swift-devel] swift in SVN is broken In-Reply-To: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070427131916.055785c0@mail.mcs.anl.gov> Message-ID: On Fri, 27 Apr 2007, Veronika V. Nefedova wrote: > Hi, > > I tried to use swift from svn trunk and the build has failed with these > errors. Please fix! Its a blocker for me... What machine were you building this on, btw? I thought you were usually using terminable, which should be giving you: $ java -version java version "1.5.0_06" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing) -- From benc at hawaga.org.uk Mon Apr 30 06:29:24 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 11:29:24 +0000 (GMT) Subject: [Swift-devel] plug points In-Reply-To: <1177892782.25695.5.camel@blabla.mcs.anl.gov> References: <1177892782.25695.5.camel@blabla.mcs.anl.gov> Message-ID: On Sun, 29 Apr 2007, Mihael Hategan wrote: > > * site selection - at least one group (the UBA people) have in part > > neglected using swift because I haven't given them an answer about how to > > do custom site selection. > > Unfortunately this is not that well documented. > http://www.cogkit.org/viewcvs/viewcvs.cgi/src/cog/modules/karajan/src/org/globus/cog/karajan/scheduler/Scheduler.java?rev=HEAD&content-type=text/vnd.viewcvs-markup There's also code in VDSScheduler.java that looks like it allows use of VDS1 site selectors (defaulting to the round robin implementation). -- From benc at hawaga.org.uk Mon Apr 30 09:50:52 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 14:50:52 +0000 (GMT) Subject: [Swift-devel] hanging workflow Message-ID: any idea why this: http://www.ci.uchicago.edu/~benc/fmri-tutorial/4-combine-img-hdr-type.swift hangs for me after running all four reslice commands? A similar (i.e. intended to run the same jobs) workflow is in http://www.ci.uchicago.edu/~benc/fmri-tutorial/3-compound-procs.swift and runs fine. -- From benc at hawaga.org.uk Mon Apr 30 10:00:20 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 15:00:20 +0000 (GMT) Subject: [Swift-devel] Re: hanging workflow In-Reply-To: References: Message-ID: On Mon, 30 Apr 2007, Ben Clifford wrote: > any idea why this: > > http://www.ci.uchicago.edu/~benc/fmri-tutorial/4-combine-img-hdr-type.swift > > hangs for me after running all four reslice commands? looks like JVM uses a bunch of CPU time while it sits hung, too. This is the end of output run with -debug. I can send the lot if desired. This happened with the same code last week or so, too, but I've only got round to playing with it in more depth today: Task(type=4, identity=urn:0-2-1-1-1177944701983) setting status to Active Task(type=4, identity=urn:0-2-1-1-1177944701983) setting status to Completed Task(type=4, identity=urn:0-0-1-2-1177944701986) setting status to Active Task(type=4, identity=urn:0-0-1-2-1177944701986) setting status to Completed reslice completed Task(type=2, identity=urn:0-3-1-1-1177944701937) setting status to Completed Staged out 4-combine-img-hdr-type-412bm8vq69yl0/shared//slice4.img to slice4.img from localhost Task(type=4, identity=urn:0-3-1-1-1177944701989) setting status to Active Task(type=4, identity=urn:0-3-1-1-1177944701989) setting status to Completed Task(type=2, identity=urn:0-1-1-1-1177944701963) setting status to Completed reslice completed Staged out 4-combine-img-hdr-type-412bm8vq69yl0/shared//slice2.img to slice2.img from localhost Task(type=4, identity=urn:0-1-1-1-1177944701992) setting status to Active Task(type=4, identity=urn:0-1-1-1-1177944701992) setting status to Completed Task(type=2, identity=urn:0-0-1-1-1177944701975) setting status to Completed reslice completed Staged out 4-combine-img-hdr-type-412bm8vq69yl0/shared//slice1.img to slice1.img from localhost Task(type=4, identity=urn:0-0-1-1-1177944701995) setting status to Active Task(type=4, identity=urn:0-0-1-1-1177944701995) setting status to Completed reslice completed -- From hategan at mcs.anl.gov Mon Apr 30 10:21:23 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 30 Apr 2007 10:21:23 -0500 Subject: [Swift-devel] Re: hanging workflow In-Reply-To: References: Message-ID: <1177946483.2979.0.camel@blabla.mcs.anl.gov> Consistently or intermittently? On Mon, 2007-04-30 at 15:00 +0000, Ben Clifford wrote: > > > > On Mon, 30 Apr 2007, Ben Clifford wrote: > > > any idea why this: > > > > http://www.ci.uchicago.edu/~benc/fmri-tutorial/4-combine-img-hdr-type.swift > > > > hangs for me after running all four reslice commands? > > looks like JVM uses a bunch of CPU time while it sits hung, too. > > This is the end of output run with -debug. I can send the lot if desired. > This happened with the same code last week or so, too, but I've only got > round to playing with it in more depth today: > > Task(type=4, identity=urn:0-2-1-1-1177944701983) setting status to Active > Task(type=4, identity=urn:0-2-1-1-1177944701983) setting status to > Completed > Task(type=4, identity=urn:0-0-1-2-1177944701986) setting status to Active > Task(type=4, identity=urn:0-0-1-2-1177944701986) setting status to > Completed > reslice completed > Task(type=2, identity=urn:0-3-1-1-1177944701937) setting status to > Completed > Staged out 4-combine-img-hdr-type-412bm8vq69yl0/shared//slice4.img to > slice4.img from localhost > Task(type=4, identity=urn:0-3-1-1-1177944701989) setting status to Active > Task(type=4, identity=urn:0-3-1-1-1177944701989) setting status to > Completed > Task(type=2, identity=urn:0-1-1-1-1177944701963) setting status to > Completed > reslice completed > Staged out 4-combine-img-hdr-type-412bm8vq69yl0/shared//slice2.img to > slice2.img from localhost > Task(type=4, identity=urn:0-1-1-1-1177944701992) setting status to Active > Task(type=4, identity=urn:0-1-1-1-1177944701992) setting status to > Completed > Task(type=2, identity=urn:0-0-1-1-1177944701975) setting status to > Completed > reslice completed > Staged out 4-combine-img-hdr-type-412bm8vq69yl0/shared//slice1.img to > slice1.img from localhost > Task(type=4, identity=urn:0-0-1-1-1177944701995) setting status to Active > Task(type=4, identity=urn:0-0-1-1-1177944701995) setting status to > Completed > reslice completed > > From benc at hawaga.org.uk Mon Apr 30 10:25:31 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 15:25:31 +0000 (GMT) Subject: [Swift-devel] Re: hanging workflow In-Reply-To: <1177946483.2979.0.camel@blabla.mcs.anl.gov> References: <1177946483.2979.0.camel@blabla.mcs.anl.gov> Message-ID: On Mon, 30 Apr 2007, Mihael Hategan wrote: > Consistently or intermittently? Entirely consistently. I've only run it on my laptop. I can try elsewhere if you're interested. Also if you give me magical incantations to make karajan output what its thinking about, that might be interesting? -- From hategan at mcs.anl.gov Mon Apr 30 10:27:16 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 30 Apr 2007 10:27:16 -0500 Subject: [Swift-devel] Re: hanging workflow In-Reply-To: References: <1177946483.2979.0.camel@blabla.mcs.anl.gov> Message-ID: <1177946836.3200.2.camel@blabla.mcs.anl.gov> On Mon, 2007-04-30 at 15:25 +0000, Ben Clifford wrote: > > On Mon, 30 Apr 2007, Mihael Hategan wrote: > > > Consistently or intermittently? > > Entirely consistently. > > I've only run it on my laptop. I can try elsewhere if you're interested. I will try it on mine. Unfortunately I don't think there's much that can be done besides starting a debugger. > > Also if you give me magical incantations to make karajan output what its > thinking about, that might be interesting? Theres 'v' 'enter' and 't' 'enter' on stdin, which should dump a sub-set of the futures which will always contain the open ones and the waiting threads respectively. But I haven't check if it works properly lately. > From benc at hawaga.org.uk Mon Apr 30 10:34:23 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 15:34:23 +0000 (GMT) Subject: [Swift-devel] Re: hanging workflow In-Reply-To: <1177946836.3200.2.camel@blabla.mcs.anl.gov> References: <1177946483.2979.0.camel@blabla.mcs.anl.gov> <1177946836.3200.2.camel@blabla.mcs.anl.gov> Message-ID: On Mon, 30 Apr 2007, Mihael Hategan wrote: > Theres 'v' 'enter' and 't' 'enter' on stdin, which should dump a sub-set > of the futures which will always contain the open ones and the waiting > threads respectively. But I haven't check if it works properly lately. doesn't seem to do anything when I type that :-( -- From benc at hawaga.org.uk Mon Apr 30 10:44:14 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 15:44:14 +0000 (GMT) Subject: [Swift-devel] Re: hanging workflow In-Reply-To: References: <1177946483.2979.0.camel@blabla.mcs.anl.gov> <1177946836.3200.2.camel@blabla.mcs.anl.gov> Message-ID: ok, so: > threads Group system: (java.lang.ref.Reference$ReferenceHandler)0x111 Reference Handler cond. waiting (java.lang.ref.Finalizer$FinalizerThread)0x110 Finalizer cond. waiting (java.lang.Thread)0x112 Signal Dispatcher running Group main: (java.lang.Thread)0x1 main cond. waiting (org.globus.cog.karajan.workflow.events.EventWorker)0x269 Worker 0 cond. waiting (org.globus.cog.karajan.workflow.events.EventWorker)0x26a Worker 1 cond. waiting (java.util.TimerThread)0x270 Timer-0 cond. waiting (org.globus.cog.karajan.workflow.events.EventDispatcher)0x274 EventDispatcher cond. waiting (java.lang.Thread)0x4fa Thread-4 running (java.lang.Thread)0x506 Thread-5 sleeping (org.griphyn.vdl.karajan.VDSAdaptiveScheduler)0x526 Scheduler cond. waiting (java.util.TimerThread)0x567 Timer-1 cond. waiting (java.lang.Thread)0x5cc pool-1-thread-3 cond. waiting (java.lang.Thread)0x5cf pool-1-thread-4 cond. waiting > The oh so delightfully names Thread-4 appears sitting in running state all the time (from typing 'threads' three times), and Thread-5 sits in sleeping state. and: Thread-4[1] where [1] java.net.PlainSocketImpl.socketAccept (native method) [2] java.net.PlainSocketImpl.accept (PlainSocketImpl.java:384) [3] java.net.ServerSocket.implAccept (ServerSocket.java:450) [4] java.net.ServerSocket.accept (ServerSocket.java:421) [5] org.griphyn.vdl.karajan.Monitor$Service.run (Monitor.java:428) [6] java.lang.Thread.run (Thread.java:613) Thread-5[1] where [1] java.lang.Thread.sleep (native method) [2] org.griphyn.vdl.karajan.InHook.run (InHook.java:54) [3] java.lang.Thread.run (Thread.java:613) Let me know what else you want... -- From yongzh at cs.uchicago.edu Mon Apr 30 10:58:48 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Mon, 30 Apr 2007 10:58:48 -0500 (CDT) Subject: [Swift-devel] hanging workflow In-Reply-To: References: Message-ID: Ben, The problem is with the array slices[] and it has long been a problem for assigning values to an array. although you assign values to its elements 0,1,2,3 but the system can not figure out how many elements the array is supposed to have (say, it could have 100 elements) thus the array is not closed and workflow hangs. The difference with the one you did run is that when you assign values to an array using syntax like: array = [elem0,elem1,elem2,elem3] Then the array is closed right after that, so the workflow continues. For Tibi's workflow, the problem is the same, values are assigned in a foreach loop, but the array is not closed after that. We talked about this a couple of times, one (not so intuitive) solution is to put the assignments into a procedure, then the array will be closed when exiting the procedure. The other way is to actively figure out when to close the array, which is not trivial. Yong. On Mon, 30 Apr 2007, Ben Clifford wrote: > any idea why this: > > http://www.ci.uchicago.edu/~benc/fmri-tutorial/4-combine-img-hdr-type.swift > > hangs for me after running all four reslice commands? > > A similar (i.e. intended to run the same jobs) workflow is in > http://www.ci.uchicago.edu/~benc/fmri-tutorial/3-compound-procs.swift > and runs fine. > > -- > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Mon Apr 30 10:59:10 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 30 Apr 2007 10:59:10 -0500 Subject: [Swift-devel] Re: hanging workflow In-Reply-To: References: <1177946483.2979.0.camel@blabla.mcs.anl.gov> <1177946836.3200.2.camel@blabla.mcs.anl.gov> Message-ID: <1177948750.4273.3.camel@blabla.mcs.anl.gov> On Mon, 2007-04-30 at 15:44 +0000, Ben Clifford wrote: > ok, so: > > > threads > Group system: > (java.lang.ref.Reference$ReferenceHandler)0x111 Reference > Handler cond. waiting > (java.lang.ref.Finalizer$FinalizerThread)0x110 Finalizer > cond. waiting > (java.lang.Thread)0x112 Signal > Dispatcher running > Group main: > (java.lang.Thread)0x1 main > cond. waiting > (org.globus.cog.karajan.workflow.events.EventWorker)0x269 Worker 0 > cond. waiting > (org.globus.cog.karajan.workflow.events.EventWorker)0x26a Worker 1 > cond. waiting > (java.util.TimerThread)0x270 Timer-0 > cond. waiting > (org.globus.cog.karajan.workflow.events.EventDispatcher)0x274 > EventDispatcher cond. waiting > (java.lang.Thread)0x4fa Thread-4 > running > (java.lang.Thread)0x506 Thread-5 > sleeping > (org.griphyn.vdl.karajan.VDSAdaptiveScheduler)0x526 Scheduler > cond. waiting > (java.util.TimerThread)0x567 Timer-1 > cond. waiting > (java.lang.Thread)0x5cc > pool-1-thread-3 cond. waiting > (java.lang.Thread)0x5cf > pool-1-thread-4 cond. waiting > > > > > The oh so delightfully names Thread-4 appears sitting in running state all > the time (from typing 'threads' three times), and Thread-5 sits in > sleeping state. Both of those are noops. One of them is the one waiting for input on stdin, and the other one is an equivalent thing waiting for a telnet connection. The workers are essential. They do the actual stuff, and they seem to be waiting (which means nothing is running). The scheduler is also sleeping. So I can't tell much from this. > > and: > > Thread-4[1] where > [1] java.net.PlainSocketImpl.socketAccept (native method) > [2] java.net.PlainSocketImpl.accept (PlainSocketImpl.java:384) > [3] java.net.ServerSocket.implAccept (ServerSocket.java:450) > [4] java.net.ServerSocket.accept (ServerSocket.java:421) > [5] org.griphyn.vdl.karajan.Monitor$Service.run (Monitor.java:428) > [6] java.lang.Thread.run (Thread.java:613) > > Thread-5[1] where > [1] java.lang.Thread.sleep (native method) > [2] org.griphyn.vdl.karajan.InHook.run (InHook.java:54) > [3] java.lang.Thread.run (Thread.java:613) > > Let me know what else you want... I wish I knew. There are two things to consider: Are there any swift/karajan threads blocked and why (the 't' 'enter' should have clarified that). Alternately you can try logging into the monitor service and type the same ('t' 'enter'). You'll find the port with netstat. > From benc at hawaga.org.uk Mon Apr 30 11:27:08 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 16:27:08 +0000 (GMT) Subject: [Swift-devel] hanging workflow In-Reply-To: References: Message-ID: On Mon, 30 Apr 2007, Yong Zhao wrote: > We talked about this a couple of times, one (not so intuitive) solution is > to put the assignments into a procedure, then the array will be closed > when exiting the procedure. OK, I just tried that. The workflow runs to completion. It looks very unpleasant for a tutorial perspective. Putting things into a procedure shouldn't cause more stuff to happen than if the procedure code was inlined. > The other way is to actively figure out when to close the array, which > is not trivial. Not trivial, but possibly what needs to happen fairly soon. -- From tiberius at ci.uchicago.edu Mon Apr 30 11:28:21 2007 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Mon, 30 Apr 2007 11:28:21 -0500 Subject: [Swift-devel] plug points In-Reply-To: References: <1177892782.25695.5.camel@blabla.mcs.anl.gov> Message-ID: The way that I read Ben's request (and I fully agree with it, I had stumbled upon the same issues and gave up because laking time to pursue them), is to get some documentation on the swift webpage for "power-users / developers" to learn how to do some basic extensions to swift (mappers, schedulers). Tibi On 4/30/07, Ben Clifford wrote: > > On Sun, 29 Apr 2007, Mihael Hategan wrote: > > > > * site selection - at least one group (the UBA people) have in part > > > neglected using swift because I haven't given them an answer about how to > > > do custom site selection. > > > > Unfortunately this is not that well documented. > > http://www.cogkit.org/viewcvs/viewcvs.cgi/src/cog/modules/karajan/src/org/globus/cog/karajan/scheduler/Scheduler.java?rev=HEAD&content-type=text/vnd.viewcvs-markup > > There's also code in VDSScheduler.java that looks like it allows use of > VDS1 site selectors (defaulting to the round robin implementation). > > -- > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > -- Tiberiu (Tibi) Stef-Praun, PhD Research Staff, Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ From benc at hawaga.org.uk Mon Apr 30 11:52:49 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 16:52:49 +0000 (GMT) Subject: [Swift-devel] plug points In-Reply-To: References: <1177892782.25695.5.camel@blabla.mcs.anl.gov> Message-ID: On Mon, 30 Apr 2007, Tiberiu Stef-Praun wrote: > The way that I read Ben's request (and I fully agree with it, I had > stumbled upon the same issues and gave up because laking time to > pursue them), is to get some documentation on the swift webpage for > "power-users / developers" to learn how to do some basic extensions to > swift (mappers, schedulers). Both concrete documentation on how the interfaces are now; but also changing the interfaces to be easy to use and coherent (for example, fairly straightforward things like bug 50 - easiness of adding mappers at run time). -- From benc at hawaga.org.uk Mon Apr 30 15:02:29 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 20:02:29 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: <1177513703.4752.20.camel@blabla.mcs.anl.gov> References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177513446.4752.15.camel@blabla.mcs.anl.gov> <1177513703.4752.20.camel@blabla.mcs.anl.gov> Message-ID: On Wed, 25 Apr 2007, Mihael Hategan wrote: > On Wed, 2007-04-25 at 10:04 -0500, Mihael Hategan wrote: > > That with the assumption that k[i-1] would work. > > It seems that k[i] = k[i-1] + i gets translated into this: > > assign(dpath="[i]", dvar=k, > > svar = sum(getFieldValue(var=k, path=concat("[]")), getFieldValue(i)) > > ) > > > > Seems like k[i-1] is translated into the wrong thing there > > (path=concat("[]")). > > It could, instead, be translated into: concat("[", subtraction(i, 1), > "]"). My guess is that would work. I tried this: [ i 1 ] which I think is pretty much what you said in the non-xml syntax. But I get this: $ swift aaaa.kml Swift V 0.0405 RunID: 7e7ozhoot4ha0 Execution failed: org.globus.cog.karajan.workflow.KarajanRuntimeException: Could not convert value to number: 1 so then I tried this instead: [ i 1 ] (adding in a getfieldvalue) and when I run that it sits in an infinite loop (second today lucky me!) Bleugh. full kml (edited from what came out of the compiler) is http://www.ci.uchicago.edu/~benc/aaaa.kml and the source I started with is in http://www.ci.uchicago.edu/~benc/aaaa.swift -- From hategan at mcs.anl.gov Mon Apr 30 15:06:40 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 30 Apr 2007 15:06:40 -0500 Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177513446.4752.15.camel@blabla.mcs.anl.gov> <1177513703.4752.20.camel@blabla.mcs.anl.gov> Message-ID: <1177963600.13173.2.camel@blabla.mcs.anl.gov> On Mon, 2007-04-30 at 20:02 +0000, Ben Clifford wrote: > > On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > On Wed, 2007-04-25 at 10:04 -0500, Mihael Hategan wrote: > > > That with the assumption that k[i-1] would work. > > > It seems that k[i] = k[i-1] + i gets translated into this: > > > assign(dpath="[i]", dvar=k, > > > svar = sum(getFieldValue(var=k, path=concat("[]")), getFieldValue(i)) > > > ) > > > > > > Seems like k[i-1] is translated into the wrong thing there > > > (path=concat("[]")). > > > > It could, instead, be translated into: concat("[", subtraction(i, 1), > > "]"). My guess is that would work. > > I tried this: > > > > > > > > [ > i > 1 > > ] > > > > > > > > > which I think is pretty much what you said in the non-xml syntax. > > But I get this: > > $ swift aaaa.kml > Swift V 0.0405 > RunID: 7e7ozhoot4ha0 > Execution failed: > org.globus.cog.karajan.workflow.KarajanRuntimeException: Could not > convert value to number: 1 That's 'i' being a vdl var. > > so then I tried this instead: > > > > > > > > > > > > > [ > > i e> > 1 > > ] > > > > > > > > > > > (adding in a getfieldvalue) > > and when I run that it sits in an infinite loop (second today lucky me!) Because of unrelated reasons I suppose. > > Bleugh. > > full kml (edited from what came out of the compiler) is > http://www.ci.uchicago.edu/~benc/aaaa.kml > > and the source I started with is in > > http://www.ci.uchicago.edu/~benc/aaaa.swift > From yongzh at cs.uchicago.edu Mon Apr 30 15:09:18 2007 From: yongzh at cs.uchicago.edu (Yong Zhao) Date: Mon, 30 Apr 2007 15:09:18 -0500 (CDT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177513446.4752.15.camel@blabla.mcs.anl.gov> <1177513703.4752.20.camel@blabla.mcs.anl.gov> Message-ID: How about ? Also do not forget the array closing business. Yong. On Mon, 30 Apr 2007, Ben Clifford wrote: > > > On Wed, 25 Apr 2007, Mihael Hategan wrote: > > > On Wed, 2007-04-25 at 10:04 -0500, Mihael Hategan wrote: > > > That with the assumption that k[i-1] would work. > > > It seems that k[i] = k[i-1] + i gets translated into this: > > > assign(dpath="[i]", dvar=k, > > > svar = sum(getFieldValue(var=k, path=concat("[]")), getFieldValue(i)) > > > ) > > > > > > Seems like k[i-1] is translated into the wrong thing there > > > (path=concat("[]")). > > > > It could, instead, be translated into: concat("[", subtraction(i, 1), > > "]"). My guess is that would work. > > I tried this: > > > > > > > > [ > i > 1 > > ] > > > > > > > > > which I think is pretty much what you said in the non-xml syntax. > > But I get this: > > $ swift aaaa.kml > Swift V 0.0405 > RunID: 7e7ozhoot4ha0 > Execution failed: > org.globus.cog.karajan.workflow.KarajanRuntimeException: Could not > convert value to number: 1 > > so then I tried this instead: > > > > > > > > > > > > > [ > > i e> > 1 > > ] > > > > > > > > > > > (adding in a getfieldvalue) > > and when I run that it sits in an infinite loop (second today lucky me!) > > Bleugh. > > full kml (edited from what came out of the compiler) is > http://www.ci.uchicago.edu/~benc/aaaa.kml > > and the source I started with is in > > http://www.ci.uchicago.edu/~benc/aaaa.swift > > -- > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From benc at hawaga.org.uk Mon Apr 30 15:17:49 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 20:17:49 +0000 (GMT) Subject: [Swift-devel] file declaration inside foreach In-Reply-To: References: <6.0.0.22.2.20070424131830.052a0af0@mail.mcs.anl.gov> <1177439552.10659.0.camel@blabla.mcs.anl.gov> <1177439953.10946.0.camel@blabla.mcs.anl.gov> <6.0.0.22.2.20070424145136.052b6ac0@mail.mcs.anl.gov> <6.0.0.22.2.20070424151503.04e12c70@mail.mcs.anl.gov> <6.0.0.22.2.20070424153029.04e11a50@mail.mcs.anl.gov> <1177447633.15483.3.camel@blabla.mcs.anl.gov> <1177513446.4752.15.camel@blabla.mcs.anl.gov> <1177513703.4752.20.camel@blabla.mcs.anl.gov> Message-ID: On Mon, 30 Apr 2007, Yong Zhao wrote: > How about > ? That seems to work. But actually, it seems like the second one that I claimed was hanging does work - maybe it was randomly hanging, maybe I was being impatient. -- From nefedova at mcs.anl.gov Mon Apr 30 15:43:09 2007 From: nefedova at mcs.anl.gov (Veronika V. Nefedova) Date: Mon, 30 Apr 2007 15:43:09 -0500 Subject: [Swift-devel] remote queue capacity Message-ID: <6.0.0.22.2.20070430153616.05f85ec0@mail.mcs.anl.gov> Hi, When I was testing today with large job submissions to TG I hit one very interesting limit -- the *total* number of jobs in the queue. Not just my own jobs in the queue - but the total queue capacity. So TG support asked if we could watch out for this message below and not submit any jobs at the moment: qsub: Maximum number of jobs already in queue For example, I could have 4 jobs in the queue and be trying to submit the 5th job, when I get that message. If the total aggregate jobs by ALL users exceeds 2048 (the number specific to TG-NCSA), I'll get that message. Can Swift be aware of such limitations - i.e. can we have some safeguards against this situation? Thanks, Nika From benc at hawaga.org.uk Mon Apr 30 15:45:25 2007 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 30 Apr 2007 20:45:25 +0000 (GMT) Subject: [Swift-devel] remote queue capacity In-Reply-To: <6.0.0.22.2.20070430153616.05f85ec0@mail.mcs.anl.gov> References: <6.0.0.22.2.20070430153616.05f85ec0@mail.mcs.anl.gov> Message-ID: On Mon, 30 Apr 2007, Veronika V. Nefedova wrote: > When I was testing today with large job submissions to TG I hit one very > interesting limit -- the *total* number of jobs in the queue. Not just my own > jobs in the queue - but the total queue capacity. So TG support asked if we > could watch out for this message below and not submit any jobs at the moment: > > qsub: Maximum number of jobs already in queue > > For example, I could have 4 jobs in the queue and be trying to submit the 5th > job, when I > get that message. If the total aggregate jobs by ALL users exceeds 2048 (the > number specific to TG-NCSA), I'll > get that message. > > Can Swift be aware of such limitations - i.e. can we have some safeguards > against this situation? is that going through GRAM or pbs directly? -- From hategan at mcs.anl.gov Mon Apr 30 16:17:42 2007 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 30 Apr 2007 16:17:42 -0500 Subject: [Swift-devel] remote queue capacity In-Reply-To: References: <6.0.0.22.2.20070430153616.05f85ec0@mail.mcs.anl.gov> Message-ID: <1177967862.13677.6.camel@blabla.mcs.anl.gov> On Mon, 2007-04-30 at 20:45 +0000, Ben Clifford wrote: > > > On Mon, 30 Apr 2007, Veronika V. Nefedova wrote: > > > When I was testing today with large job submissions to TG I hit one very > > interesting limit -- the *total* number of jobs in the queue. Not just my own > > jobs in the queue - but the total queue capacity. So TG support asked if we > > could watch out for this message below and not submit any jobs at the moment: > > > > qsub: Maximum number of jobs already in queue > > > > For example, I could have 4 jobs in the queue and be trying to submit the 5th > > job, when I > > get that message. If the total aggregate jobs by ALL users exceeds 2048 (the > > number specific to TG-NCSA), I'll > > get that message. > > > > Can Swift be aware of such limitations - i.e. can we have some safeguards > > against this situation? > > is that going through GRAM or pbs directly? GRAM seems to report the following: "It is unknown if the job was submitted". This is GRAM error code 126 and that's pretty much all the client gets. >