<div>Thank you all for your replies ...</div>
<div> </div>
<div>I have a workflow that i developed using C code. I am thinking of using Swift to execute the workflow, so my thinking is that i need first to change the code to be Swift script.</div>
<div> </div>
<div>More info about my workflow:</div>
<div> </div>
<div>The workflow consist of several major tasks:</div>
<div> </div>
<div>Task 1: create a 1000 uniquely strings where each string is 1000 bytes.</div>
<div>Task 2: merge the strings where every 2 strings ( A, B) will exchange a segment of it at a certain point and produce 1 string (C) with the same length (1000 bytes). Then C replaces B.</div>
<div>Task 3: duplicate the list of string so we will have now 2000 strings</div>
<div>Task 4: randomly choose 1000 strings from the current 2000 strings. </div>
<div>Task 5: repeat Tasks 2, 3, and 4 for N times (N is given) and now the list of strings used in the next iteration is the output of Task 4.</div>
<div> </div>
<div> </div>
<div>Do you think changing the whole program into Swift script is necessary or just certain sections? Can i just use wrappers around certain tasks and use Swift Script to call these tasks?</div>
<div> </div>
<div>Will the performance be the same?</div>
<div> </div>
<div> </div>
<div>Any suggestion will be really appreciated.</div>
<div> </div>
<div>Thanks,</div>
<div>Jamal</div>
<div> </div>
<div> </div>
<div><br><br> </div>
<div><span class="gmail_quote">On 6/3/08, <b class="gmail_sendername">Ioan Raicu</b> <<a href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a>> wrote:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div text="#000000" bgcolor="#ffffff">Hi,<br>There are several papers out there from our group that shows different aspects of performance of Swift. Here are a few:<br>
<ul>
<li><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://people.cs.uchicago.edu/%7Eiraicu/publications/2008_NOVA08_book-chapter_Swift.pdf" target="_blank">http://people.cs.uchicago.edu/~iraicu/publications/2008_NOVA08_book-chapter_Swift.pdf</a>
<ul>
<li>Figure 9: Shows the memory footprint per job (aka tasks, or nodes in the DAG graph)
<ul>
<li>shows memory footprint of 3.2KB per node </li></ul>
<li>Figure 18: Shows a large scale application
<ul>
<li>20K tasks on 200 CPUs with average task lengths of 200 seconds is a comfortable range for Swift and Falkon
<li>we have more recent results, not published yet, that has 16K tasks on 2048 CPUs with an average task length of 87 seconds which worked well<br></li></li></ul></li></li></ul>
<li><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://people.cs.uchicago.edu/%7Eiraicu/publications/2007_SWF07_Swift.pdf" target="_blank">http://people.cs.uchicago.edu/~iraicu/publications/2007_SWF07_Swift.pdf</a>
<ul>
<li>Figure 6: Shows the speedup achieved with different task lengths
<ul>
<li>Conclusion is that using multi-level scheduling with the Falkon provider, even tasks in the range of seconds long can achieve good speedup<br></li></ul>
<li>Figure 7: Shows the throughput in tasks/sec achieved by Swift
<ul>
<li>shows Swift achieving 50+ tasks/sec throughputs using Falkon<br>
<li>the paragraph right after this figure mentions that Swift running directly with GRAM2 and PBS can achieve 2 jobs/sec; the implication of this is that jobs typically take 15~60 seconds to startup, which reflects the cost of scheduling, scheduling cycles, and local resource manager's (LRM) time to setup the remote nodes; there are also limitations on how many jobs can be submitted at a time, as each job queued might consume some resources on the LRM, or there might be policies in place that limit the number of jobs that can be queued; this means that aggressive throttling must take place, which in practice, reduces the sustained rate that Swift can submit/execute jobs to a single site, to even lower than 2 jobs/sec </li>
</li></ul></li></li></ul></li></li></ul>So, to answer you question, the performance of Swift (and any other workflow system) will heavily rely on how efficient you can dispatch jobs/tasks to remote resources, how long jobs/tasks are, how data intensive the application is, and how much data movement must happen before the job runs and after. If you have a fast enough file system, and your application execution times are small, you can expect anywhere from 1 to 50 jobs/sec from Swift, depending on what technologies you use to interface between Swift and the remote resources (e.g. GRAM, PBS, Condor, Falkon, etc).<br>
<br>Cheers,<br>Ioan <br><br><br>J A wrote:
<blockquote type="cite">
<div><span class="e" id="q_11a4f2e92997eaca_1">
<div>Hi All:</div>
<div> </div>
<div>
<p style="MARGIN: 0in 0in 0pt"><font face=""><font size="2">Based on my reading, the performance from execution a swift workflow depends on the parallelism<span> </span>that a workflow has.</font></font></p>
<p style="MARGIN: 0in 0in 0pt"><font face="" size="2"> </font></p>
<p style="MARGIN: 0in 0in 0pt"><font face=""><font size="2">If I have a workflow that contains several processors where each processor (procedure) depends on the previous one (output of a processor "A" is the input for processors "B" and so on.)</font></font></p>
<p style="MARGIN: 0in 0in 0pt"><font face="" size="2"> </font></p>
<p style="MARGIN: 0in 0in 0pt"><font face=""><font size="2">How the performance of using swift in this case compare to other systems that execute workflows where there isn't any parallelism in the workflow?</font></font></p>
<p style="MARGIN: 0in 0in 0pt"><font face="" size="2"> </font></p>
<p style="MARGIN: 0in 0in 0pt"><font face="" size="2">-- <br>Thanks,<br><span></span><span><br>Jamal </span></font></p></div></span></div><pre><hr width="90%" size="4"><span class="q">
_______________________________________________
Swift-user mailing list
<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:Swift-user@ci.uchicago.edu" target="_blank">Swift-user@ci.uchicago.edu</a>
<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-user" target="_blank">http://mail.ci.uchicago.edu/mailman/listinfo/swift-user</a>
</span></pre></blockquote><br><pre cols="72">--
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: <a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:iraicu@cs.uchicago.edu" target="_blank">iraicu@cs.uchicago.edu</a>
Web: <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.cs.uchicago.edu/%7Eiraicu" target="_blank">http://www.cs.uchicago.edu/~iraicu</a>
<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://dev.globus.org/wiki/Incubator/Falkon" target="_blank">http://dev.globus.org/wiki/Incubator/Falkon</a>
<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page" target="_blank">http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page</a>
===================================================
===================================================
</pre></div></blockquote></div><br>