[Swift-devel] swift-on-ec2

[Ben Clifford]

On Tue, 15 May 2007, Ioan Raicu wrote:

> If we can get the data caching working in Falkon, we might be able to 
> run Swift over Falkon without a shared file system.  This is still work 
> in progress, but we might be closer to achieving this that not.  BTW, 
> the data caching would mean that Swift does not stage in any data 
> anymore, but wold essentially stand up a GridFTP server from where 
> Falkon workers would get the needed data just when they need it.  We are 
> still ironing out all this stuff, but it could potentially do away with 
> the shared file sytem assumption.

In the longer term, Swift possibly won't have its input data on the 
submitting system - for example, if data is mapped from remote gridftp 
servers, then it should be transferred directly from those ftp servers to 
the execute side (perhaps to a shared filesystem, perhaps direct to a 
worker node), and output data should be transferred back fairly directly, 
rather than going via the submit system.

If Falkon is doing its own 'interesting' data movement stuff, then it 
would probably be a good idea for it to mesh in with what Swift (eg. swift 
provides a list of stage-these-in and stage-these-out URLs or something 
like that and has various ways of performing that, such as submitting a 
transfer job, or passing that information onto falkon)

--