[Swift-devel] problems with external dependencies

[Mihael Hategan]

I'm curious what you are doing with the directory after you wait for it.

On a different note, should there be an array size function (e.g.
size(array)), it would have to wait for the array to be closed before
giving an accurate answer.

On Sun, 2009-03-22 at 19:38 -0500, Michael Wilde wrote:
> I got the example from my previous email working using this technique 
> (passing the external var to trace). But a script that simulates more 
> closely what I really need to do is still eluding me.
> 
> In the real code, I need to wait till a set of nested procedures that 
> involve nested foreach and iterate statements complete. So Im trying 
> create a simple simulation of the needed synchronization with the 
> following script:
> 
> --
> type file;
> 
> app (file o) echo (int i) { echo i stdout=@o; }
> 
> (file r[]) generate() {
>    int j[] = [0:10];
>    foreach i in j {
>      r[i] = echo(i*i);
>    }
> }
> 
> (external w) wait(file dir[]) {
>    trace("in wait: dir",dir);
> }
> 
> app (file o) ls (string dir, external w) { ls "-l" dir stdout=@o; }
> 
> file datadir[]<simple_mapper;prefix="datadir/">;
> datadir = generate();
> 
> external w1 = wait(datadir);
> 
> trace( "generate done", w1);
> 
> file out <"ls.out">;
> out = ls("/home/wilde/oops/swift/datadir/", w1);
> --
> 
> In this script the proc "generate()" simulates the production of the 
> data directory. I want the proc "ls" which simulates the processing of 
> the data directory, to wait until the directory is produced. As the 
> directory has too many files to pass to "ls" as an array, I pass a 
> string with the dir's path to ls, and want external vars to cause it to 
> wait till the directory is complete.
> 
> But in the case above, returning the dataset (file array) "datadir" from 
> generate() does not wait for the array to be "closed". Nor does passing 
> it to wait(), nor does passing it by name to trace().  The script gives:
> 
> --
> Swift svn swift-r2724 (swift modified locally) cog-r2333
> 
> RunID: 20090322-1922-o4ibjxac
> Progress:
> 
> SwiftScript trace: in wait: dir, 
> org.griphyn.vdl.karajan.FuturePairIterator at 1e671e67
> 
> SwiftScript trace: generate done, null
> 
> Progress:  Selecting site:8 Active:1 Stage out:1 Failed:1 Finished 
> successfully:1
> Progress:  Selecting site:4 Active:1 Stage out:1 Failed:1 Finished 
> successfully:5
> Progress:  Active:1 Stage out:1 Failed:1 Finished successfully:9
> Final status:  Failed:1 Finished successfully:11
> 
> The following errors have occurred:
> 1. Application "ls" failed (Exit code 2)
>          Arguments: "-l, /home/wilde/oops/swift/datadir/"
>          Host: localhost
>          Directory: ex5-20090322-1922-o4ibjxac/jobs/l/ls-ljbsob8j
>          STDERR: /bin/ls: /home/wilde/oops/swift/datadir/: No such file 
> or directory
>          STDOUT:
> --
> 
> 
> It seems there's only 2 kinds of constructs or behaviors that can give 
> me this behavior, neither of which I can find a way to cause:
> - something that waits for the whole array to get its values
> - something that waits for an entire array of externals to all be set
> 
> This note in the users guide suggests a possible way to do what I need:
> 
> "Statements which deal with the array as a whole will often wait for the 
> array to be closed before executing (thus, a closed array is the 
> equivalent of a non-array type being assigned). However, a foreach 
> statement will apply its body to elements of an array as they become 
> known. It will not wait until the array is closed.
> 
> What statement can I use to "wait for the array to be closed before 
> executing"?
> 
> 
> On 3/22/09 4:47 PM, Ben Clifford wrote:
> > As far as I can tell from a brief poke around, this is what is happening 
> > for you:
> > 
> > Compound procedures do not themselves wait for their input parameters 
> > to all be ready to use. instead, they start trying to run all component 
> > pieces.
> > 
> > If some data necessary for some component piece is not ready yet, that 
> > component piece will wait, so the compound procedure doesn't need to (and 
> > indeed shouldn't, because that reduces potential parallelism in some 
> > cases)
> > 
> > You say this:
> > 
> > analyseDatabase(external i) {                                                   
> >   trace("i am analyseDatabase");                                                
> > }   
> > 
> > The trace call does not have any need to wait for i to be ready. So it 
> > doesn't wait for i to be ready.
> > 
> > If you say this:
> >                                                                                 
> > analyseDatabase(external i) {
> >   trace("i am analyseDatabase", i);
> > }
> > 
> > then the trace call must wait for i to be ready (and fortuitously in the 
> > present implementation doesn't explode even though i cannot be 
> > meaningfully traced).
> > 
> > With that change, you'll see the behaviour you want.
> > 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel