[Swift-devel] Re: Question of wrapper.sh

[Ben Clifford]

On Fri, 7 Mar 2008, Ioan Raicu wrote:

> symbolic linking.  Since the mkdir is much improved now, I assume that is not
> the bottleneck, but doing 10~20 echo to a log file on the shared file system
> from many nodes at the same time is expensive, which I think is the main
> bottleneck in the current wrapper script.  Once Zhao is done disabling all
> logging, except for necessary ones, we'll have a better idea of how fast we
> can go, and if it is necessary to eliminate the mkdir step as well. 

When I was playing with this around the time of SC, I put in a bunch of 
progress logging inside the wrapper script. This adds to the amount of 
logging that the wrapper does, but gives a many stage breakdown of where 
the wrapper script is spending its time.

Run a bunch of jobs, eg a few thousand, with latest SVN and 
wrapperlog.always.transfer=true set in swift.properties.

You'll get a <runid>.d directory, with a bunch of .info files. From there 
I (or you) can graph how each wrapper script spent its time.

Ideally there should be a bunch of steps taking almost no time, then the 
executable, then another bunch of steps taking almost no time; but doing 
this should reveal wrong behaviour there.

Poke me when you have that dump directory and I can have a look.

--