[Swift-devel] transfers of small files

[Mihael Hategan]

So I've been playing with that issue. I've made some measurements
outside Swift. Here's a summary:

32k files. From terminable to tg-uc

1 - karajan with connection caching. transfers in parallel. tops at
200KB/s

2 - n*globus-url-copy - With 32 parallel transfers it starts failing and
gets about 10KB/s

3 - globus-url-copy with a list of files: around 300KB/s

4 - globus-url-copy with a list of files, E mode, and data channel
re-use: 500KB/s

So I figured I should hack the GridFTP provider to re-use data channels
by default. This is where it gets strange. I get averages (over multiple
runs) of over 1MB/s, with mins of about 130KB and max of 1.9MB/s, but
with a lot of variability. I'll debug this. However, I think there is
still value in enabling this by default.