[Swift-devel] problems with external dependencies

[Michael Wilde]

On 3/22/09 10:14 PM, Mihael Hategan wrote:
> I'm curious what you are doing with the directory after you wait for it.

Walking through the files in that dir, checking simulation outputs for 
convergence, and determining the new protein secondary structure for the 
next round of simulations if they have not yet converged.

> On a different note, should there be an array size function (e.g.
> size(array)), it would have to wait for the array to be closed before
> giving an accurate answer.

Ive run into the need/desire for a len(array) function several times.

That might indeed one way to code the wait, albeit a tad idiomatic.

A waitForClose() function may also be useful, on which you could base 
length().

Also note that the limitation that forced me into this approach was the 
inability to pass a dataset without invoking all the semantics of data 
transport and the attendant cmd line args required. Some way to do that 
may also be worth discussing.


> On Sun, 2009-03-22 at 19:38 -0500, Michael Wilde wrote:
>> I got the example from my previous email working using this technique 
>> (passing the external var to trace). But a script that simulates more 
>> closely what I really need to do is still eluding me.
>>
>> In the real code, I need to wait till a set of nested procedures that 
>> involve nested foreach and iterate statements complete. So Im trying 
>> create a simple simulation of the needed synchronization with the 
>> following script:
>>
>> --
>> type file;
>>
>> app (file o) echo (int i) { echo i stdout=@o; }
>>
>> (file r[]) generate() {
>>    int j[] = [0:10];
>>    foreach i in j {
>>      r[i] = echo(i*i);
>>    }
>> }
>>
>> (external w) wait(file dir[]) {
>>    trace("in wait: dir",dir);
>> }
>>
>> app (file o) ls (string dir, external w) { ls "-l" dir stdout=@o; }
>>
>> file datadir[]<simple_mapper;prefix="datadir/">;
>> datadir = generate();
>>
>> external w1 = wait(datadir);
>>
>> trace( "generate done", w1);
>>
>> file out <"ls.out">;
>> out = ls("/home/wilde/oops/swift/datadir/", w1);
>> --
>>
>> In this script the proc "generate()" simulates the production of the 
>> data directory. I want the proc "ls" which simulates the processing of 
>> the data directory, to wait until the directory is produced. As the 
>> directory has too many files to pass to "ls" as an array, I pass a 
>> string with the dir's path to ls, and want external vars to cause it to 
>> wait till the directory is complete.
>>
>> But in the case above, returning the dataset (file array) "datadir" from 
>> generate() does not wait for the array to be "closed". Nor does passing 
>> it to wait(), nor does passing it by name to trace().  The script gives:
>>
>> --
>> Swift svn swift-r2724 (swift modified locally) cog-r2333
>>
>> RunID: 20090322-1922-o4ibjxac
>> Progress:
>>
>> SwiftScript trace: in wait: dir, 
>> org.griphyn.vdl.karajan.FuturePairIterator at 1e671e67
>>
>> SwiftScript trace: generate done, null
>>
>> Progress:  Selecting site:8 Active:1 Stage out:1 Failed:1 Finished 
>> successfully:1
>> Progress:  Selecting site:4 Active:1 Stage out:1 Failed:1 Finished 
>> successfully:5
>> Progress:  Active:1 Stage out:1 Failed:1 Finished successfully:9
>> Final status:  Failed:1 Finished successfully:11
>>
>> The following errors have occurred:
>> 1. Application "ls" failed (Exit code 2)
>>          Arguments: "-l, /home/wilde/oops/swift/datadir/"
>>          Host: localhost
>>          Directory: ex5-20090322-1922-o4ibjxac/jobs/l/ls-ljbsob8j
>>          STDERR: /bin/ls: /home/wilde/oops/swift/datadir/: No such file 
>> or directory
>>          STDOUT:
>> --
>>
>>
>> It seems there's only 2 kinds of constructs or behaviors that can give 
>> me this behavior, neither of which I can find a way to cause:
>> - something that waits for the whole array to get its values
>> - something that waits for an entire array of externals to all be set
>>
>> This note in the users guide suggests a possible way to do what I need:
>>
>> "Statements which deal with the array as a whole will often wait for the 
>> array to be closed before executing (thus, a closed array is the 
>> equivalent of a non-array type being assigned). However, a foreach 
>> statement will apply its body to elements of an array as they become 
>> known. It will not wait until the array is closed.
>>
>> What statement can I use to "wait for the array to be closed before 
>> executing"?
>>
>>
>> On 3/22/09 4:47 PM, Ben Clifford wrote:
>>> As far as I can tell from a brief poke around, this is what is happening 
>>> for you:
>>>
>>> Compound procedures do not themselves wait for their input parameters 
>>> to all be ready to use. instead, they start trying to run all component 
>>> pieces.
>>>
>>> If some data necessary for some component piece is not ready yet, that 
>>> component piece will wait, so the compound procedure doesn't need to (and 
>>> indeed shouldn't, because that reduces potential parallelism in some 
>>> cases)
>>>
>>> You say this:
>>>
>>> analyseDatabase(external i) {                                                   
>>>   trace("i am analyseDatabase");                                                
>>> }   
>>>
>>> The trace call does not have any need to wait for i to be ready. So it 
>>> doesn't wait for i to be ready.
>>>
>>> If you say this:
>>>                                                                                 
>>> analyseDatabase(external i) {
>>>   trace("i am analyseDatabase", i);
>>> }
>>>
>>> then the trace call must wait for i to be ready (and fortuitously in the 
>>> present implementation doesn't explode even though i cannot be 
>>> meaningfully traced).
>>>
>>> With that change, you'll see the behaviour you want.
>>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>