[Swift-devel] swift-on-ec2

[Tim Freeman]

On Wed, 16 May 2007 11:55:18 -0500
Ioan Raicu <iraicu at cs.uchicago.edu> wrote:

> Hi,
> I am just catching up with emails from last night...
> 
> Ben Clifford wrote:
> > On Tue, 15 May 2007, Kate Keahey wrote:
> >
> >   
> >> As Ian says, Borja and I were planning to meet with Ioan on Thursday to 
> >> discuss interaction between Falkon and the workspace service (not 
> >> necessarily/exclusively in the EC2 context). I don't completely 
> >> understand the relationship between swift and falkon -- are there 
> >> specific applications or scenarios that you are trying to target in this 
> >> exercise?
> >>     
> >
> > By virtue of the fact that they come from pretty much the same group of 
> > people, they're somewhat fuzzily related - but pretty much swift is 
> > generating (over the duration of its execution, rather than in one batch) 
> > a bunch of jobs that need executing (as well, as various things like file 
> > transfers). As it generates them, it sends them off to be executed. The 
> > official ways that are 'supported' by Swift are by executing them on the 
> > local machine and by sending them off through GRAM; however, people can 
> > plug in whatever they want to do submissions.
> >
> > I know less about Falkon because it isn't Swift, but the Falkon side of 
> > things is pretty much about running a bunch of jobs - it plugs into the 
> > abovementioned place in Swift so that Swift gives Falkon jobs to run, and 
> > Falkon runs them (with a goal of Falkon being, presumably, to run it much 
> > more efficiently than if they were submitted straight through GRAM - it 
> > seems to do pretty well).
> >   
> We intentionally made Falkon's interface and semantics as similar as 
> possible to that of GRAM, so applications that normally used GRAM could 
> easily change to Falkon.
> > There's two things going on with swift - one is about making it 
> > straightforward to use at the low end of things, so that people can start 
> > using it easily - for the most part, that isn't interesting in itself; the 
> > other is about getting it to perform well at the high end of things, which 
> > is where the fun research is. Using Falkon and using EC2 are both on that 
> > side of things.
> >   
> Right! 
> 
> Falkon is certainly about getting more performance from the same hardware. 
> 
> EC2 on the other hand is more about a new paradigm of how resources are 
> acquired.  In the batch-scheduled world, the demand for resources is 
> usually higher than the supply.  In EC2, its likely that the supply for 
> resources is higher than the demand.  With that said, it means that with 
> EC2, it is likely that you could always get more resources now if you 
> were willing to pay for them

That's not entirely true at this particular point in time:

http://www.pcworld.com/article/id,130832-c,webservices/article.html

"We hate being capacity-constrained," Bezos said. "It's not the right way to
run a business. We are trying to get ourselves in a position with EC2 where we
will be demand-constrained instead of capacity-constrained."


> ... this could have implications on the 
> resource allocation and management policies that govern when it makes 
> sense to get more resources and when not to.

Right now for example, we're programming a little feature into the workspace-EC2
gateway that limits the amount of money an entity can spend :-) 

Tim 


>   Using EC2 might be about 
> performance, but the really interesting part that I see emerging is a 
> new model that deviates from the traditional batch-scheduled systems the 
> Grid community has grown accustomed to.
> 
> Ioan