[Swift-devel] swift-on-ec2

[Kate Keahey]

Ioan,

Ioan Raicu wrote:
> Well, the dynamic provisioning assumes that Falkon is acquiring 
> resources when it needs them.  This implies that it knows how to talk to 
> the EC2 service, and it knows how to bootstrap a VM that has the 
> necessary Falkon software stack.
> 
> I was actually hoping (at least in the short term) that static resource 
> provisioning could be handled by the workspace service, talking to the 
> EC2 service and bootstraping the VM (with the necesarry Falkon stack), 
> and then once the Falkon executors register with the Falkon dispatcher, 
> then Falkon handles the lightweight job management (in place of a 
> traditional LRM).

Yes, this is exactly what I was also thinking. My point below is that 
the combined infrastructure would fit into the swift.

> The provisioning to EC2 could be pushed onto Falkon in the future, but 
> it is not currently on my immediate list of things to-do list.
> 
> Ioan
> 
> Kate Keahey wrote:
>> Thanks Ben, this helps a lot! So it seems to me like we are talking 
>> about combining dynamic provisioning with lightweight job management 
>> which should be pluggable into swift.
>>
>> Ben Clifford wrote:
>>> On Tue, 15 May 2007, Kate Keahey wrote:
>>>
>>>> As Ian says, Borja and I were planning to meet with Ioan on Thursday 
>>>> to discuss interaction between Falkon and the workspace service (not 
>>>> necessarily/exclusively in the EC2 context). I don't completely 
>>>> understand the relationship between swift and falkon -- are there 
>>>> specific applications or scenarios that you are trying to target in 
>>>> this exercise?
>>>
>>> By virtue of the fact that they come from pretty much the same group 
>>> of people, they're somewhat fuzzily related - but pretty much swift 
>>> is generating (over the duration of its execution, rather than in one 
>>> batch) a bunch of jobs that need executing (as well, as various 
>>> things like file transfers). As it generates them, it sends them off 
>>> to be executed. The official ways that are 'supported' by Swift are 
>>> by executing them on the local machine and by sending them off 
>>> through GRAM; however, people can plug in whatever they want to do 
>>> submissions.
>>>
>>> I know less about Falkon because it isn't Swift, but the Falkon side 
>>> of things is pretty much about running a bunch of jobs - it plugs 
>>> into the abovementioned place in Swift so that Swift gives Falkon 
>>> jobs to run, and Falkon runs them (with a goal of Falkon being, 
>>> presumably, to run it much more efficiently than if they were 
>>> submitted straight through GRAM - it seems to do pretty well).
>>>
>>> There's two things going on with swift - one is about making it 
>>> straightforward to use at the low end of things, so that people can 
>>> start using it easily - for the most part, that isn't interesting in 
>>> itself; the other is about getting it to perform well at the high end 
>>> of things, which is where the fun research is. Using Falkon and using 
>>> EC2 are both on that side of things.
>>>
>>
> 

-- 

Kate Keahey,
Mathematics & CS Division, Argonne National Laboratory
Computation Institute, University of Chicago