[Swift-devel] transfers of small files

[Mihael Hategan]

By contrast, multiple large files are transferred at a max of 11MB/s in
(1).

On Wed, 2007-11-28 at 18:20 -0600, Mihael Hategan wrote:
> So I've been playing with that issue. I've made some measurements
> outside Swift. Here's a summary:
> 
> 32k files. From terminable to tg-uc
> 
> 1 - karajan with connection caching. transfers in parallel. tops at
> 200KB/s
> 
> 2 - n*globus-url-copy - With 32 parallel transfers it starts failing and
> gets about 10KB/s
> 
> 3 - globus-url-copy with a list of files: around 300KB/s
> 
> 4 - globus-url-copy with a list of files, E mode, and data channel
> re-use: 500KB/s
> 
> So I figured I should hack the GridFTP provider to re-use data channels
> by default. This is where it gets strange. I get averages (over multiple
> runs) of over 1MB/s, with mins of about 130KB and max of 1.9MB/s, but
> with a lot of variability. I'll debug this. However, I think there is
> still value in enabling this by default.
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>