[Swift-devel] fakecnari on ranger without gridftp

[Ben Clifford]

I have an app 'fakecnari' that behaves somewhat like the CNARI app that 
skenny has been working on, in order to make it easier for me to look at 
bottlenecks.

So far, that's had similar problems to skenny's real runs where input 
files cannot be staged in to ranger fast enough from UC - this limits the 
number of cores that can be used at any one time on Ranger to around 15.

So I thought in order to see what other bottlenecks might be found, I'd 
make a run with swift running directly on a ranger headnode, submitting 
through coasters and with the input and output files moved around using 
the local copy file provider (the same as happens when you use the default 
local site).

This looks like it manages to use over 100 cores quite a lot. The speedup 
for the run is (including allocation time for coaster workers, which is 
a significant part of this run) about 50000s worth of sleep done in 800s, 
which is sleeping 62 times as fast as on a single core. Discounting 
worker allocation time, this takes about 590s which is sleeping about 84 
times as fast.

Even with local copies instead of ftp, file transfers (limited to 4 at 
once) appear to be a rate limiting factor.

There are full plots here:

http://www.ci.uchicago.edu/~benc/tmp/report-fakecnari-20080928-1134-herl17vf/

--