[Swift-devel] swift-on-ec2

[Ioan Raicu]

That would make things much simpler, from Falkon's perspective.  
Essentially, if the workspace service offered an interface that Falkon 
to allocate and de-allocate resources (VMs) on demand, then the Falkon 
dynamic resource provisioning could be used as long as Falkon implement 
this new workspace interface instead of the current GRAM interface it 
uses!  Then, the whole EC2 deployment and bootstrapping would be 
offloaded to the worspace service, and only the resource provisioning 
and task dispatch would be done at Falkon, the same as it is today when 
we use GRAM!

Ioan

Yong Zhao wrote:
> I'd think the the workspace manager should be able to do that, and not
> statically, but allocate new virtual nodes as requested.
>
> Yong.
>
> On Wed, 16 May 2007, Ioan Raicu wrote:
>
>   
>> Well, the dynamic provisioning assumes that Falkon is acquiring
>> resources when it needs them.  This implies that it knows how to talk to
>> the EC2 service, and it knows how to bootstrap a VM that has the
>> necessary Falkon software stack.
>>
>> I was actually hoping (at least in the short term) that static resource
>> provisioning could be handled by the workspace service, talking to the
>> EC2 service and bootstraping the VM (with the necesarry Falkon stack),
>> and then once the Falkon executors register with the Falkon dispatcher,
>> then Falkon handles the lightweight job management (in place of a
>> traditional LRM).
>>
>> The provisioning to EC2 could be pushed onto Falkon in the future, but
>> it is not currently on my immediate list of things to-do list.
>>
>> Ioan
>>
>> Kate Keahey wrote:
>>     
>>> Thanks Ben, this helps a lot! So it seems to me like we are talking
>>> about combining dynamic provisioning with lightweight job management
>>> which should be pluggable into swift.
>>>
>>> Ben Clifford wrote:
>>>       
>>>> On Tue, 15 May 2007, Kate Keahey wrote:
>>>>
>>>>         
>>>>> As Ian says, Borja and I were planning to meet with Ioan on Thursday
>>>>> to discuss interaction between Falkon and the workspace service (not
>>>>> necessarily/exclusively in the EC2 context). I don't completely
>>>>> understand the relationship between swift and falkon -- are there
>>>>> specific applications or scenarios that you are trying to target in
>>>>> this exercise?
>>>>>           
>>>> By virtue of the fact that they come from pretty much the same group
>>>> of people, they're somewhat fuzzily related - but pretty much swift
>>>> is generating (over the duration of its execution, rather than in one
>>>> batch) a bunch of jobs that need executing (as well, as various
>>>> things like file transfers). As it generates them, it sends them off
>>>> to be executed. The official ways that are 'supported' by Swift are
>>>> by executing them on the local machine and by sending them off
>>>> through GRAM; however, people can plug in whatever they want to do
>>>> submissions.
>>>>
>>>> I know less about Falkon because it isn't Swift, but the Falkon side
>>>> of things is pretty much about running a bunch of jobs - it plugs
>>>> into the abovementioned place in Swift so that Swift gives Falkon
>>>> jobs to run, and Falkon runs them (with a goal of Falkon being,
>>>> presumably, to run it much more efficiently than if they were
>>>> submitted straight through GRAM - it seems to do pretty well).
>>>>
>>>> There's two things going on with swift - one is about making it
>>>> straightforward to use at the low end of things, so that people can
>>>> start using it easily - for the most part, that isn't interesting in
>>>> itself; the other is about getting it to perform well at the high end
>>>> of things, which is where the fun research is. Using Falkon and using
>>>> EC2 are both on that side of things.
>>>>
>>>>         
>> --
>> ============================================
>> Ioan Raicu
>> Ph.D. Student
>> ============================================
>> Distributed Systems Laboratory
>> Computer Science Department
>> University of Chicago
>> 1100 E. 58th Street, Ryerson Hall
>> Chicago, IL 60637
>> ============================================
>> Email: iraicu at cs.uchicago.edu
>> Web:   http://www.cs.uchicago.edu/~iraicu
>>        http://dsl.cs.uchicago.edu/
>> ============================================
>> ============================================
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>>     
>
>   

-- 
============================================
Ioan Raicu
Ph.D. Student
============================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
============================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
       http://dsl.cs.uchicago.edu/
============================================
============================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20070516/f4bb5d69/attachment.html>