[Swift-user] Performance of Swift

[Ioan Raicu]

Hi,
There are several papers out there from our group that shows different 
aspects of performance of Swift.  Here are a few:

    * http://people.cs.uchicago.edu/~iraicu/publications/2008_NOVA08_book-chapter_Swift.pdf
          o Figure 9: Shows the memory footprint per job (aka tasks, or
            nodes in the DAG graph)
                + shows memory footprint of 3.2KB per node
          o Figure 18: Shows a large scale application
                + 20K tasks on 200 CPUs with average task lengths of 200
                  seconds is a comfortable range for Swift and Falkon
                + we have more recent results, not published yet, that
                  has 16K tasks on 2048 CPUs with an average task length
                  of 87 seconds which worked well
    * http://people.cs.uchicago.edu/~iraicu/publications/2007_SWF07_Swift.pdf
          o Figure 6: Shows the speedup achieved with different task lengths
                + Conclusion is that using multi-level scheduling with
                  the Falkon provider, even tasks in the range of
                  seconds long can achieve good speedup
          o Figure 7: Shows the throughput in tasks/sec achieved by Swift
                + shows Swift achieving 50+ tasks/sec throughputs using
                  Falkon
                + the paragraph right after this figure mentions that
                  Swift running directly with GRAM2 and PBS can achieve
                  2 jobs/sec; the implication of this is that jobs
                  typically take 15~60 seconds to startup, which
                  reflects the cost of scheduling, scheduling cycles,
                  and local resource manager's (LRM) time to setup the
                  remote nodes; there are also limitations on how many
                  jobs can be submitted at a time, as each job queued
                  might consume some resources on the LRM, or there
                  might be policies in place that limit the number of
                  jobs that can be queued; this means that aggressive
                  throttling must take place, which in practice, reduces
                  the sustained rate that Swift can submit/execute jobs
                  to a single site, to even lower than 2 jobs/sec

So, to answer you question, the performance of Swift (and any other 
workflow system) will heavily rely on how efficient you can dispatch 
jobs/tasks to remote resources, how long jobs/tasks are, how data 
intensive the application is, and how much data movement must happen 
before the job runs and after.  If you have a fast enough file system, 
and your application execution times are small, you can expect anywhere 
from 1 to 50 jobs/sec from Swift, depending on what technologies you use 
to interface between Swift and the remote resources (e.g. GRAM, PBS, 
Condor, Falkon, etc).

Cheers,
Ioan


J A wrote:
> Hi All:
>  
>
> Based on my reading, the performance from execution a swift workflow 
> depends on the parallelism that a workflow has.
>
>  
>
> If I have a workflow that contains several processors where each 
> processor (procedure) depends on the previous one (output of a 
> processor "A" is the input for processors "B" and so on.)
>
>  
>
> How the performance of using swift in this case compare to other 
> systems that execute workflows where there isn't any parallelism in 
> the workflow?
>
>  
>
> -- 
> Thanks,
>
> Jamal
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>   

-- 
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
http://dev.globus.org/wiki/Incubator/Falkon
http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page
===================================================
===================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20080603/73779a77/attachment.html>