[Swift-devel] Fwd: [Swift-user] Specifying execution dependencies directly

[Ian Foster]

This would be a nice thing to have in the Swift manual: "how to do mapreduce problems"

Begin forwarded message:

> From: Mihael Hategan <hategan at mcs.anl.gov>
> Date: February 6, 2010 12:48:41 PM CST
> To: Andriy Fedorov <fedorov at bwh.harvard.edu>
> Cc: swift-user at ci.uchicago.edu
> Subject: Re: [Swift-user] Specifying execution dependencies directly
> 
> On Sat, 2010-02-06 at 13:03 -0500, Andriy Fedorov wrote:
>> Hi,
>> 
>> I have the following (what seems to me, typical) analysis scenario. I
>> am running multiple instances of task A, task B needs to wait until
>> all of the instances of A are completed, and analyze the outputs
>> produced by As.
>> 
>> If I was running this locally, I would pass the name of the directory
>> with the results to B. However, in Swift, I need to somehow define
>> that B cannot start until all of the As complete. Since B would expect
>> a directory name, or an arbitrarily long list of files, I am not sure
>> how this can be done. Should I develop the workflow as a shell script
>> that would run swift script with As, then run B locally, then proceed
>> with the rest of the computation?
> 
> You are right, this is a typical map/reduce problem.
> 
> Use an array that holds the results of all the As and pass that to B.
> 
> file aout[];
> foreach v, k in someRangeOrSomething {
>  aout[k] = a(v);
> }
> file bout;
> bout = b(aout);
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100207/16d85648/attachment.html>