[Swift-devel] Re: Another performance comparison of DOCK

[Michael Wilde]

Ben, can you point me to the graphs for this run? (Zhao's *99cy0z4g.log)

Here's a high-level summary of this run:

Swift end   16:42:17
Swift start 16:09:07
Runtime        33:10 = 1990 seconds

2048 cores

Total app wall time = 1190260 seconds

1190260 / ( 1990 * 2048 ) = .29 efficiency

Once stage-ins start to complete, are the corresponding jobs initiated 
quickly, or is Swift doing mostly stage-ins for some period?

Zhao indicated he saw data indicating there was about a 700 second lag 
from workflow start time till the first Falkon jobs started, if I 
understood correctly. Do the graphs confirm this or say something different?

If the 700-second delay figure is true, and stage-in was eliminated by 
copying input files right to the /tmp workdir rather than first to 
/shared, then we'd have:

1190260 / ( 1290 * 2048 ) = .45 efficiency

A good gain, but only partway to a number that looks good.

I assume we're paying the same staging price on the output side?

What I think we learned from the MARS app run, which had no input data 
and only tiny output data files (10 bytes vs 10K bytes), was that the 
optimized wrapper achieved somewhere between .7 to .8 efficiency.

I'd like to look at whatever data we can get from this or similar 
subsequent runs to learn what steps we could take next to increase the 
efficiency metric.  Guidance welcome.

Thanks,

Mike


On 4/13/08 11:37 AM, Ben Clifford wrote:
> On Sat, 12 Apr 2008, Zhao Zhang wrote:
> 
>> Hi, Ben
>>
>> I got a log file of 6084 successful runs on BGP. Check it here,
>> terninable:/home/zzhang/swift_file/dock2-20080412-1609-99cy0z4g.log
> 
> This one runs better - it gets up to a peak of 5000 jobs submitted into 
> Falkon simultaneously, and spends a considerable amount of time over the 
> 2048 level that I suppose is what you need to be over to get all 2048 CPUs 
> used.
> 
> There's a lot of stage-in activity that probably could be eliminated / 
> changed for the single-filesytem case.
> 


- Mike