[Swift-devel] swift-on-ec2

[Mihael Hategan]

I think we're moving towards a scenario in which Falkon does
increasingly more things that it wasn't supposed to do. That includes
scheduling and data management (which, is a tricky business if we look
at the necessity for throttling, error handling and other management
issues).
I'm not sure if this is a good idea from an engineering standpoint.

Mihael

On Tue, 2007-05-15 at 23:24 +0000, Ben Clifford wrote:
> On Tue, 15 May 2007, Ioan Raicu wrote:
> 
> > If we can get the data caching working in Falkon, we might be able to 
> > run Swift over Falkon without a shared file system.  This is still work 
> > in progress, but we might be closer to achieving this that not.  BTW, 
> > the data caching would mean that Swift does not stage in any data 
> > anymore, but wold essentially stand up a GridFTP server from where 
> > Falkon workers would get the needed data just when they need it.  We are 
> > still ironing out all this stuff, but it could potentially do away with 
> > the shared file sytem assumption.
> 
> In the longer term, Swift possibly won't have its input data on the 
> submitting system - for example, if data is mapped from remote gridftp 
> servers, then it should be transferred directly from those ftp servers to 
> the execute side (perhaps to a shared filesystem, perhaps direct to a 
> worker node), and output data should be transferred back fairly directly, 
> rather than going via the submit system.
> 
> If Falkon is doing its own 'interesting' data movement stuff, then it 
> would probably be a good idea for it to mesh in with what Swift (eg. swift 
> provides a list of stage-these-in and stage-these-out URLs or something 
> like that and has various ways of performing that, such as submitting a 
> transfer job, or passing that information onto falkon)
>