[Swift-devel] Re: Data transfer test results

[Michael Wilde]

I think in general they are larger, but I will investigate.

In the meantime, run173 just finished, staging in data from the tp-osg 
server.  This went very nice: staged in 1000 45MB files in 10 minutes!

I think yesterday I measured the disk-to-disk copy time using dd of a 
2.5GB file at 2 minutes, so this WAN transfer from CI to Argonne at 10 
minutes is only about 2.5X slower.  Thats not bad, and 10 minutes to 
stage the whole dataset is not bad.

Lets discuss net how best to achieve or simulate/hack caching of inputs 
on the local site.  Whats the best way to do that and test it?

- Mike

(btw - run173 above failed in the end, I think, due to long cmd line 
length.  Need to discuss that as well, as we may need to demo a 
summarization job such as this test simulates).

On 11/7/07 12:58 PM, Ben Clifford wrote:
> 
> On Wed, 7 Nov 2007, Michael Wilde wrote:
> 
>> I ran a swift script that invoked one app with the same 1000 45MB input files
>> that angle-1000 reads.
> 
> 45mb files don't seem particularly representative of the data - I picked 
> spool_190 at random and see that the average file size is 14mb but that 
> many are small, in the 20kb range.
> 
> Tuning for 45mb files is likely to not be the same as tuning for 23kb 
> files.
>