[Swift-devel] swift-on-ec2

[Ioan Raicu]

One of the 2 main motivations for Falkon was the data management.  We 
saw early on that we need to couple the compute and data resource 
management, and that is what we are doing as we push forward with 
Falkon.  Falkon should be something that could be usable by other 
applications, that don't have all the smarts of Swift, that simply want 
to run jobs efficiently and have the data management abstracted away. 

The main idea is that Swift's data management will likely still be 
needed (at a site level), but Falkon can push that further to the 
physical node level.  Swift and Falkon will likely evolve independently, 
but if we work together, we can ensure that they can inter-operate, as 
thy do today!

Ioan

Mihael Hategan wrote:
> I think we're moving towards a scenario in which Falkon does
> increasingly more things that it wasn't supposed to do. That includes
> scheduling and data management (which, is a tricky business if we look
> at the necessity for throttling, error handling and other management
> issues).
> I'm not sure if this is a good idea from an engineering standpoint.
>
> Mihael
>
> On Tue, 2007-05-15 at 23:24 +0000, Ben Clifford wrote:
>   
>> On Tue, 15 May 2007, Ioan Raicu wrote:
>>
>>     
>>> If we can get the data caching working in Falkon, we might be able to 
>>> run Swift over Falkon without a shared file system.  This is still work 
>>> in progress, but we might be closer to achieving this that not.  BTW, 
>>> the data caching would mean that Swift does not stage in any data 
>>> anymore, but wold essentially stand up a GridFTP server from where 
>>> Falkon workers would get the needed data just when they need it.  We are 
>>> still ironing out all this stuff, but it could potentially do away with 
>>> the shared file sytem assumption.
>>>       
>> In the longer term, Swift possibly won't have its input data on the 
>> submitting system - for example, if data is mapped from remote gridftp 
>> servers, then it should be transferred directly from those ftp servers to 
>> the execute side (perhaps to a shared filesystem, perhaps direct to a 
>> worker node), and output data should be transferred back fairly directly, 
>> rather than going via the submit system.
>>
>> If Falkon is doing its own 'interesting' data movement stuff, then it 
>> would probably be a good idea for it to mesh in with what Swift (eg. swift 
>> provides a list of stage-these-in and stage-these-out URLs or something 
>> like that and has various ways of performing that, such as submitting a 
>> transfer job, or passing that information onto falkon)
>>
>>     
>
>
>   

-- 
============================================
Ioan Raicu
Ph.D. Student
============================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
============================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
       http://dsl.cs.uchicago.edu/
============================================
============================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20070516/1dd7b778/attachment.html>