[Swift-devel] swift-on-ec2

[Ioan Raicu]

Ben Clifford wrote:
> On Tue, 15 May 2007, Ioan Raicu wrote:
>
>   
>> If we can get the data caching working in Falkon, we might be able to 
>> run Swift over Falkon without a shared file system.  This is still work 
>> in progress, but we might be closer to achieving this that not.  BTW, 
>> the data caching would mean that Swift does not stage in any data 
>> anymore, but wold essentially stand up a GridFTP server from where 
>> Falkon workers would get the needed data just when they need it.  We are 
>> still ironing out all this stuff, but it could potentially do away with 
>> the shared file sytem assumption.
>>     
>
> In the longer term, Swift possibly won't have its input data on the 
> submitting system - for example, if data is mapped from remote gridftp 
> servers, then it should be transferred directly from those ftp servers to 
> the execute side (perhaps to a shared filesystem, perhaps direct to a 
> worker node), and output data should be transferred back fairly directly, 
> rather than going via the submit system.
>   
Right, from Falon's point of view, this would not be any different than 
having the GridFTP server at the submit host. 
> If Falkon is doing its own 'interesting' data movement stuff, then it 
> would probably be a good idea for it to mesh in with what Swift (eg. swift 
> provides a list of stage-these-in and stage-these-out URLs or something 
> like that and has various ways of performing that, such as submitting a 
> transfer job, or passing that information onto falkon)
>   
The idea is to do just this!  Get Swift to pass in its normal URLs of 
input and output data, and then have Falkon do its own data management 
using those URLs!  The idea is to not change anything fundamental in 
Swift, but ensure that enough information is passed to Falkon so it can 
operate properly, and do its own data management! 

Ioan

-- 
============================================
Ioan Raicu
Ph.D. Student
============================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
============================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
       http://dsl.cs.uchicago.edu/
============================================
============================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20070515/2af84c8b/attachment.html>