[Swift-devel] [Fwd: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)]

[Mihael Hategan]

On Mon, 2009-10-19 at 00:32 -0500, Mihael Hategan wrote:
> -------- Forwarded Message --------
> From: Ben Clifford <benc at hawaga.org.uk>
> To: hategan at mcs.anl.gov
> Subject: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)
> Date: Mon, 19 Oct 2009 03:28:07 +0000 (GMT)
> 
> i sent this earlier then i remembered that no one seems to 
> moderate-approve my swift devel postings ever...

I don't seem to be getting those.

On this topic, why did we unsubscribe one of swift's committers from the
swift mailing list(s)?

> 
> ---------- Forwarded message ----------
> Date: Mon, 19 Oct 2009 01:59:44 +0000 (GMT)
> From: Ben Clifford <benc at hawaga.org.uk>
> To: swift-devel at ci.uchicago.edu
> Subject: re: IO overheads of swift wrapper scripts on BlueGene/P
> 
> 
> swift r2970 changed copying outputs to a hard-link of outputs to improve 
> performance at the output stage. you might be interested to see how that 
> works.
> 
> (also, don't use the same runid for different runs. you'll get into 
> trouble somehow sooner or later. )
> 
> 
> r2970 | benc | 2009-06-22 23:40:53 +1000 (Mon, 22 Jun 2009) | 6 lines
> 
> Make wrapper use mv to stage out files to shared directory rather than cp.
> This should be faster when the job directories are on the same filesystem
> as the site shared cache.
> 
> (suggested by tuecke)


Right. We're already using that. mv, on the same fs, would just make an
entry in the new directory pointing to the same data instead of
duplicating the data. Though I suppose this depends on the FS.

However, we have to filesystems here, so it doesn't apply.