[Swift-devel] lots of very small files vs gridftp

[Mihael Hategan]

On Wed, 2008-10-01 at 00:02 +0000, Ben Clifford wrote:
> On Tue, 30 Sep 2008, Mihael Hategan wrote:
> 
> > But I'm not otherwise opposed to anything in particular. I suppose
> > taring/untaring could be done manually, at the expense of messing the
> > abstractness of swift.
> 
> I played some making Swift do tar/untar of stageins automatically (so no 
> modifications are needed to the SwiftScript code).

This reminds me of clustering of jobs we did initially with swift
versus Falkon.

> 
> Theres a plot here 
> http://www.ci.uchicago.edu/~benc/tmp/report-fakecnari-20080930-1820-0nmtamxg/
> 
> Basically the first 600s are taken up allocating coaster workers, and the 
> remaining time uses quite a lot of cores at once. So the total duration of 
> run doesn't seem that different; but I think that the behaviour as number 
> of jobs increases will be better- the 600s startup is a fixed cost (which 
> I also think can be massively reduced in a couple of ways) and the bit 
> that is proportional to the number of jobs is the remaining three hundred 
> seconds.

Could the untar job be done with fork straight through gram?

> 
> 
> This is a fairly dirty hack - there's no clustering for stageouts;

Though that could be done in a similar way, right?

>  there 
> is fairly crude decision of whether to cluster transfers or not 
> (basically, queue file transfers for 30s and after that, if there's more 
> than one, make a cluster).
> 
> The initial startup is slow, I think, because the initial startup of 
> coaster workers is done based on a malformed job submission caused by the 
> low quality of this clustering code - it doesn't pass through the 
> coastersPerNode parameter for initial jobs so the initial coaster worker 
> is very slow.

Using fork would probably solve this, too.

Now, not to leave the other side of the argument on its own, running
other fileops (mkdir, ls, etc.) through coasters does offer the added
benefit of parallelizing the RTT. We do at least one such operation per
job. In the 2^16 jobs case, and with parallelism of 2^7 vs. 2^3, one
would get, in the theoretical case, an improvement of (2^16/2^3 -
2^16/2^7)RTTs. Or (8192 - 512)RTTs. Or 7680s for an RTT of 1s.

But then that could probably also be done in GridFTP with pipelining.