[Swift-devel] Clustering and Temp Dirs with Swift

[Ben Clifford]

On Sat, 27 Oct 2007, Michael Wilde wrote:

> I suspect that my angle workflow on UC teragrid was having similar problems:
> lots of jobs finishing but data coming back very slowly.
> (Btw I really appreciate everyones efforts on this and I *do* realize that its
> a weekend)

Is this the one that looks like you were hitting the maximum-of-4-at-once 
limit on file transfers?

> Ben: is the log_processing code changing as we speak, and is it sensible for
> me and others to try to run your latest versions? Or just send you logfiles?

It always changes. But you can svn update whenever you want.

If you put a log file (and associated kickstart records) in the usual 
repository then its easy enough for me to run the code on it.

> Question: do people feel that a move to local disk could be done 
> *entirely* in wrapper.sh, or is it known that other parts of swift would 
> have to change as well?

I think that there won't be a trivial solution to this problem. At 
present, the model is quite strongly tied to a site-shared filesystem (as 
VDS was before).

In the past, we've discussed informally different ways of moving data 
round between submit-side storage locations, site-wide storage locations 
and worker-local storage. I think this is another use case for that; but I 
think the general conclusion that that's a non-trivial thing to do is 
still valid.

> For the moment, until I hear comments on the questions above, I will 
> work on Angle, see if I get the same problems (I should see the same) 
> and try to start a simple text doc on the data management mechanism that 
> will at least help *me* better understand whats going on.

For angle, a first thing to try is increasing the transfer throttle.

If there's lock contention there, it may be that will decrease, rather 
than increase the performance.

--