[Swift-user] Question about arrays and ordering of execution

[Lorenzo Pesce]

Thanks Mike.

It looks like it is working. Now we need to scale it up (and then adds some more twists...). Medium scale tests planned for today (up to 50-100 nodes).

If this works we have only problem type II and then we are ready to plan the seminar.  :-)


On Mar 22, 2013, at 2:17 PM, Michael Wilde wrote:

> 
> Lorenzo,
> 
>> I start with having a number of files associated with a specific
>> Sample, do something with all of them, then merge them and move on
>> to do more things (which also involved splitting them and remerging
>> them).
>> 
>> My rough swift code for it would be the following.
>> 
>> string [] FILES = readdata.... ;
>> 
>> string {} OUTFILES = <some mapper >; # don't exist yet
>> 
>> for file in FILES {
>> 
>> (OUTFILE{file}...) do_something (file);
>> }
>> 
>> (next_file) = do_somethingmore(OUTFILE);
>> 
>> 
>> My question is: can I use an array/hash of files as input to a
>> function and will this be sufficient to tell swift to wait till
>> *all* the OUTFILE elements have been run?
> 
> By "merge them" do you mean "merge them all into one file"?
> 
> If do_somethingmore is the merge function, and its an app(), it will wait for OUTFILE to be closed before it runs.
> 
> There are very few situations in which you need to tell swift to wait: that should happen transparently: if you refer to a data element (ie, a scalar, array element, array, etc) with an app or a primitive, swift will wait for that element to have a value.
> 
> If OUTFILE is very large, however, you may need to use some special techniques to pass all the file names to an app. (I think that involves using writeData() to write the file names to another file and pass the file). Im not sure thats well tested or sufficiently documented yet, but lets deal with that after you get the basic structure of the script worked out.
> 
> Another thing to watch out for is whether string-indexed arrays is handled correctly by all mappers. Would it be useful for you to work through the logic of a simple case first?  Id suggest to write up the example above to handle 10 files, first. I think you can find this logic in the MODIS example in the 2011 Swift Language paper.
> 
> - Mike
> 
>> If not, which is the next simplest approach (most portable, less
>> prone to fall to pieces, less dependent on filesystems and so on).
>> 
>> Thanks a lot as usual,
>> 
>> Lorenzo
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>