[Swift-devel] Re: IO overheads of swift wrapper scripts on BlueGene/P

[Michael Wilde]

Cool, Allan, thanks for the update. Especially for our CDM work, it 
would be useful to go back and figure out where the lengthy delays were 
coming from, as that helps justify CDM. I suspect its contention in the 
creation of jobdirs on shared storage, or contention within a jobdir, or 
both. We'll want to see how this overhead grows with scale; it doesnt 
seem to take much parallelism to cause bad degradation with the default 
data management.

- Mike


On 10/23/09 4:14 PM, Allan Espinosa wrote:
> Hello,
> 
> I just tried  Mike's swift build in
> /home/falkon/swift/src/cog/modules/swift/dist/swift-svn/bin/swift .
> My workflow ran for 838 seconds.  This is pretty comparable to a
> falkon-only run of 829 seconds.
> 
> I guess the patches (local CN jobdir) fixes things.
> 
> thanks Mike!
> 
> -Allan
> 
> 2009/10/16 Allan Espinosa <aespinosa at cs.uchicago.edu>:
>> The attached graph looks interesting. The green on the left shows the
>> 192 job workload using the straight falkon client. The right one is
>> through swift.  Using pure falkon the total time for the workload is
>> at around 814 seconds while the swift workflow took 1520 seconds.  In
>> some repeats of the same run, Swift took 3078 seconds.
>>