[Swift-devel] Clustering and Temp Dirs with Swift

[Ben Clifford]

On Fri, 26 Oct 2007, Mihael Hategan wrote:

> On GPFS, according to my understanding of their documentation, exactly
> one node controls access to one file at any given time. If, for all
> observable aspects of the implementation, a directory is a file with a
> bunch of metadata for the files it contains, then doing things in a
> directory from multiple places is similar to accessing the same file
> from multiple places.

Googling around, the IBM document 'Sizing and Tuning GPFS' talks about 
this a small amount http://www.redbooks.ibm.com/redbooks/pdfs/sg245610.pdf

Its for RS/6000 (but I think that's irrelevant) and I don't know how it 
ties in versionwise (which is more relevant)

I think it agrees with what you say above.

They briefly discuss write contention for a directory page 62, section 
2.4.2.

I think basically they're saying in that section that what we do with our 
shared directories is going to not work very well, because of contention 
for the write lock on the big (i.e. lots of people accessing) shared 
directories.

It might even be said that the site directory layout at the moment is 
perfectly designed to work badly with GPFS' directory model, in that there 
is a lot of shared directory use (for status reports, data file caching, 
etc) even when jobs are entirely independent.

However, this GPFS behaviour, if it really is whats causing the problem, 
is possibly relatively straightforward to accomodate.

I think its worth spending effort accomodating GPFS, given its scalability 
on other axes.

--