[Swift-devel] transfers of small files

[Ian Foster]

Mihael:

It isn't clear to me--are you using the "lots of small files" 
optimization here?

I've CCed John Bresnahan so he can comment.

Ian.

Mihael Hategan wrote:
> So I've been playing with that issue. I've made some measurements
> outside Swift. Here's a summary:
>
> 32k files. From terminable to tg-uc
>
> 1 - karajan with connection caching. transfers in parallel. tops at
> 200KB/s
>
> 2 - n*globus-url-copy - With 32 parallel transfers it starts failing and
> gets about 10KB/s
>
> 3 - globus-url-copy with a list of files: around 300KB/s
>
> 4 - globus-url-copy with a list of files, E mode, and data channel
> re-use: 500KB/s
>
> So I figured I should hack the GridFTP provider to re-use data channels
> by default. This is where it gets strange. I get averages (over multiple
> runs) of over 1MB/s, with mins of about 130KB and max of 1.9MB/s, but
> with a lot of variability. I'll debug this. However, I think there is
> still value in enabling this by default.
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>   

-- 

   Ian Foster, Director, Computation Institute
Argonne National Laboratory & University of Chicago
Argonne: MCS/221, 9700 S. Cass Ave, Argonne, IL 60439
Chicago: Rm 405, 5640 S. Ellis Ave, Chicago, IL 60637
Tel: +1 630 252 4619.  Web: www.ci.uchicago.edu.
      Globus Alliance: www.globus.org.