[Swift-devel] [Fwd: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)]

[Michael Wilde]

On 10/19/09 12:39 AM, Mihael Hategan wrote:
> On Mon, 2009-10-19 at 00:32 -0500, Mihael Hategan wrote:
>> -------- Forwarded Message --------
>> From: Ben Clifford <benc at hawaga.org.uk>
>> To: hategan at mcs.anl.gov
>> Subject: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)
>> Date: Mon, 19 Oct 2009 03:28:07 +0000 (GMT)
>>
>> i sent this earlier then i remembered that no one seems to 
>> moderate-approve my swift devel postings ever...

I do, when I see them.

> 
> I don't seem to be getting those.
> 
> On this topic, why did we unsubscribe one of swift's committers from the
> swift mailing list(s)?

I thought this was done by Ben. He's certainly welcome to subscribe.

> 
>> ---------- Forwarded message ----------
>> Date: Mon, 19 Oct 2009 01:59:44 +0000 (GMT)
>> From: Ben Clifford <benc at hawaga.org.uk>
>> To: swift-devel at ci.uchicago.edu
>> Subject: re: IO overheads of swift wrapper scripts on BlueGene/P
>>
>>
>> swift r2970 changed copying outputs to a hard-link of outputs to improve 
>> performance at the output stage. you might be interested to see how that 
>> works.
>>
>> (also, don't use the same runid for different runs. you'll get into 
>> trouble somehow sooner or later. )
>>
>>
>> r2970 | benc | 2009-06-22 23:40:53 +1000 (Mon, 22 Jun 2009) | 6 lines
>>
>> Make wrapper use mv to stage out files to shared directory rather than cp.
>> This should be faster when the job directories are on the same filesystem
>> as the site shared cache.
>>
>> (suggested by tuecke)
> 
> 
> Right. We're already using that. mv, on the same fs, would just make an
> entry in the new directory pointing to the same data instead of
> duplicating the data. Though I suppose this depends on the FS.
> 
> However, we have to filesystems here, so it doesn't apply.
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel