[Swift-devel] several alternatives to design the data management system for Swift on SuperComputers

[Ioan Raicu]

Mihael Hategan wrote:
> On Mon, 2008-12-01 at 16:52 -0600, Ioan Raicu wrote:
>
>   
> ...
>
>   
>>>   
>>>       
>> I don't think you realize how expensive GPFS access is when doing so
>> at 100K CPU scale.
>>     
>
> I don't think I understand what you mean by "access". As I said, things
> that generate contention are going to be slow.
>
> If the problem requires that contention to happen, then it doesn't
> matter what the solution is. If it does not, then I suspect that there
> is a way to avoid contention in GPFS, too (sticking things in different
> directories).
>   
The basic idea is that many smaller shared file systems will scale 
better than 1 large file system, as the contention is localized. The 
problem is that having 1 global namespace is simple and straight 
forward, but having N local namespaces is not, and requires extra 
management. If we try to beat GPFS by implementing a global 
shared/parallel file system, we are likely to fail, as Ben or you 
already mentioned... but by changing the landscape a bit by splitting 
the single large global space into many smaller spaces, we should gain 
scalability and performance larger scales.

Ioan
>
>
>   

-- 
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
http://dev.globus.org/wiki/Incubator/Falkon
http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page
===================================================
===================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20081201/325147d9/attachment.html>