[Swift-devel] several alternatives to design the data management system for Swift on SuperComputers

[Ioan Raicu]

Ben Clifford wrote:
> On Mon, 1 Dec 2008, Zhao Zhang wrote:
>
>   
> ...
>> 3. Collective IO
>>   Build a shared file system which could be accessed by all CN, instead of
>> writing output data to GPFS, workers
>>   copy intermediate output data to this shared ram-disk. And retrieve the data
>> from IFS.
>>
>>   Several Concerns:
>>   a) reliability of torus network --- we need to test more about this.
>>   b) performance of torus network --- could this be really performing better
>> than GPFS? If not, at what scale
>>       could torus perform better than GPFS?
>>     
>
> As phrased above, this seems a little strange:
>
> "rather than use a shared file system, lets build a shared file system and 
> use it."
>   
Zhao meant that instead of using 1 large global shared file system, lets 
build many smaller shared file systems, that can keep traffic localized 
within the machine (i.e. per pset) to allow relatively linear 
scalability with every additional processing power and network 
bandwidth. Depending at what level we are working at, we might also be 
able to relax some of the semantics of the POSIX file system, in order 
to improve performance and scalability, at the expense of other things.

Ioan
> Do you mean building some general purpose posix shared file system? If so, 
> this seems quite hard, and seems directly in competition with PVFS and 
> GPFS, a competition which you are pretty much guaranteed to lose.
>
> It may be that you mean something completely different - your concerns 
> about the torus network seem unrelated to writing a posix fs, so I think 
> that may be the case (or maybe you are overspecialising your concerns).
>
>   

-- 
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
http://dev.globus.org/wiki/Incubator/Falkon
http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page
===================================================
===================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20081201/62eada5f/attachment.html>