<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">This would be a nice thing to have in the Swift manual: "how to do mapreduce problems"<br><div><br><div>Begin forwarded message:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>From: </b></span><span style="font-family:'Helvetica'; font-size:medium;">Mihael Hategan <<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>Date: </b></span><span style="font-family:'Helvetica'; font-size:medium;">February 6, 2010 12:48:41 PM CST<br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>To: </b></span><span style="font-family:'Helvetica'; font-size:medium;">Andriy Fedorov <<a href="mailto:fedorov@bwh.harvard.edu">fedorov@bwh.harvard.edu</a>><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>Cc: </b></span><span style="font-family:'Helvetica'; font-size:medium;"><a href="mailto:swift-user@ci.uchicago.edu">swift-user@ci.uchicago.edu</a><br></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, 1);"><b>Subject: </b></span><span style="font-family:'Helvetica'; font-size:medium;"><b>Re: [Swift-user] Specifying execution dependencies directly</b><br></span></div><br><div>On Sat, 2010-02-06 at 13:03 -0500, Andriy Fedorov wrote:<br><blockquote type="cite">Hi,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I have the following (what seems to me, typical) analysis scenario. I<br></blockquote><blockquote type="cite">am running multiple instances of task A, task B needs to wait until<br></blockquote><blockquote type="cite">all of the instances of A are completed, and analyze the outputs<br></blockquote><blockquote type="cite">produced by As.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">If I was running this locally, I would pass the name of the directory<br></blockquote><blockquote type="cite">with the results to B. However, in Swift, I need to somehow define<br></blockquote><blockquote type="cite">that B cannot start until all of the As complete. Since B would expect<br></blockquote><blockquote type="cite">a directory name, or an arbitrarily long list of files, I am not sure<br></blockquote><blockquote type="cite">how this can be done. Should I develop the workflow as a shell script<br></blockquote><blockquote type="cite">that would run swift script with As, then run B locally, then proceed<br></blockquote><blockquote type="cite">with the rest of the computation?<br></blockquote><br>You are right, this is a typical map/reduce problem.<br><br>Use an array that holds the results of all the As and pass that to B.<br><br>file aout[];<br>foreach v, k in someRangeOrSomething {<br> aout[k] = a(v);<br>}<br>file bout;<br>bout = b(aout);<br><br><br>_______________________________________________<br>Swift-user mailing list<br><a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>http://mail.ci.uchicago.edu/mailman/listinfo/swift-user<br></div></blockquote></div><br></body></html>