[Swift-devel] swift-on-ec2

[Ben Clifford]

Ian asked about this elsewhere, but its perhaps interesting for 
swift-devel people to look at the questions too.

On Tue, 15 May 2007, Ian Foster wrote:

> Dear All:
                                                                                
> I asked Kate if she and Tim could look into creating VM images that 
> would allow us to run Swift applications on Amazon EC2. I think Kate is 
> meeting with Ioan about this on Thursday (?).
                                                                                
> One issue that I thought would be good to discuss is what we'd want in 
> that VM image. Perhaps this is obvious to the rest of you, but it isn't 
> to me. A few thoughts:

> * I'm assuming that we want to run "workers" on EC2 nodes, and have the "task
> dispatch" logic run on some external frontend system outside EC2.

> * I would think that we want to use Falkon to do the task dispatch. If so, we
> need a Falkon executor on each VM, configured to check in with the Falkon
> dispatcher. (Alternatively, we could use, say, SGE: in that case, we would
> want an SGE agent.)

> *  We need a way of getting data to and from the worker nodes. Do we want to
> run a file system across the EC2 nodes and the external frontend node? That
> seems rather inefficient. Other options?

> * Should we preload the application code on each EC2 node?

Here's a couple of approaches:

 1) swift regards all the EC2 nodes that we are paying for as a single 
    site.

Something like falkon handles all the task dispatch and worker node 
management. I don't know what that looks like at the moment in Falkon, but 
the interface for Swift to send jobs into Falkon sounds pretty 
straightforward and shouldn't need changing.

All the nodes in a site are required by our site model to have a shared 
filesystem - we've talked about removing it, but I think that is still the 
case and if so, isn't going to change soon. timf probably knows more than 
the people on this list about making shared filesystems.

In this case, falkon would be doing the site selection.

 2) swift regards each EC2 node as a separate site.

So Swift would be doing site selection between each site (i.e. between 
each EC2 node), and then submitting to that site.

I don't know if the interface between Swift and eg. Falkon allows swift to 
tell Falkon which remote node to run on.

However, Swift would then be able to use something like gridftp to stage 
to each EC2 node (assuming that EC2 nodes can act as ftp servers - I don't 
know what their network connectivity is like) - a shared filesystem 
between all nodes in a site is pretty simple when there is only a single 
node in the site.


Amazon also has a storage cloud, alongside its compute cloud. I know very 
little about that and have never thought about how it would fit into the 
above (if at all). Maybe someone else knows more.

--