[Swift-devel] several alternatives to design the data management system for Swift on SuperComputers

[Mihael Hategan]

On Mon, 2008-12-01 at 17:10 -0600, Ioan Raicu wrote:
> 
> 
> Mihael Hategan wrote: 
> > On Mon, 2008-12-01 at 16:52 -0600, Ioan Raicu wrote:
> > 
> >   
> > ...
> > 
> >   
> > > I don't think you realize how expensive GPFS access is when doing so
> > > at 100K CPU scale.
> > >     
> > 
> > I don't think I understand what you mean by "access". As I said, things
> > that generate contention are going to be slow.
> > 
> > If the problem requires that contention to happen, then it doesn't
> > matter what the solution is. If it does not, then I suspect that there
> > is a way to avoid contention in GPFS, too (sticking things in different
> > directories).
> >   
> The basic idea is that many smaller shared file systems will scale
> better than 1 large file system, as the contention is localized.

Which is the same behaviour you get if you have a hierarchy of
directories. This is what Ben implemented in Swift.

>  The problem is that having 1 global namespace is simple and straight
> forward, but having N local namespaces is not, and requires extra
> management.

Right. That's why most filesystems I know of treat directories as
independent files containing file metadata (aka. "local namespaces").