[Swift-devel] GPFS issue of SWIFT on BGP

[Ben Clifford]

On Tue, 10 Feb 2009, Zhao Zhang wrote:

> What if there are 640 ssh-providers sending result files at the same time? Do
> you know any successful test case
> with hundreds of ssh-providers working together with one submit host?

Control is the other way round.

The Swift client will pull files down from the I/O nodes when jobs are 
finished. (that is done by the dostageout call in execute2 in 
libexec/vdl-int.k)

Swift has rate limiting on the number of file transfers and file 
operations that can be in progress at any one time. By default, the limit 
is 4 (for file transfers) and 8 (for file operations). This is controlled 
by the throttle.transfers and throttle.file.operations settings in 
swift.properties.

I think (but I am not sure) that this is a limit for the whole of Swift, 
rather than per site (but I am not sure).

If jobs are finishing faster than Swift can stage out the data (which is 
likely to happen) then a queue of transfer requests will build up inside 
Swift.

I think it is quite likely (though I have no numerical evidence) that you 
will find provider-ssh copies files too slowly for your liking; in which 
case you would need to come up with a faster way of moving files between 
the IO nodes and the submitting node. But you should see what happens with 
provider-ssh first. You should easily be able to compute throughput rates 
when you have log files for this.

--