[Swift-devel] discussion for swift output file path

[Ben Clifford]

On Thu, 6 Nov 2008, Mihael Hategan wrote:

> Only if we assume non-reentrant read locks. And this would need some
> investigation, but if concurrent reads do not create contention, then
> there is no problem, because once a job is done, there is no more
> writing to its directory.

If job output files are left in their job directory, rather than put in 
some hashed-by-filename shared directory, I think that is true. Otherwise, 
the access patterns for any given hash bucket for a hash-on-filename are 
going to be fairly random, with reads and writes happening at all sorts of 
different times and from many different places.

Keeping output files in job-specific directories seems a reasonable thing 
to do performance wise, but means more awareness on the submit side about 
which files came from which job.

--