[Swift-devel] fakecnari on ranger without gridftp

[Ben Clifford]

On Sun, 28 Sep 2008, Ian Foster wrote:

> Zhao and others have been working a lot on BG/P, as you probably know. The
> challenges inherent in file transfers there are in some ways comparable. Their
> solution has been to develop collective I/O operations that (a) multicast
> replicated input files more efficiently than multiple independent reads, and
> (b) apply two-stage methods to bundled inputs and outputs, moving them from
> local disks to the global file system via an intermediate file system. I
> wonder whether similar methods could be applied here?

yes, I imagine methods that help in falkon+bg/p will help here too, at 
least in the abstract.

> Independently of that, it would be useful to develop a performance model for
> the whole end-to-end system to determine where the bottlenecks are, and the
> peak performance that could be expected for each stage (including data
> movement) so we can see where there are opportunities for improvement, and
> where we are limited by fundamental limits.

right.

certainly I know that transferring the same amount of data as a tarball in 
this particular app seems much much faster (9s vs 450s) - see Subject: 
[Swift-devel] lots of very small files vs gridftp.

>From a raw CPU perspective, there's 65535 parallel tasks each of a few 
seconds long each; there's a fairly obvious, if naive, target there of 65k 
x speedup (mmm).

I still don't really have a feel for what the filesystem on Ranger 
(lustre) will do - its been behaving fairly well so far, I think, but I 
imagine thats because it hasn't been terribly heavily loaded in my 
testing.

--