[Swift-devel] Re: Question of wrapper.sh

[Ioan Raicu]

Ben Clifford wrote:
> On Fri, 7 Mar 2008, Ioan Raicu wrote:
>
>   
>> symbolic linking.  Since the mkdir is much improved now, I assume that is not
>> the bottleneck, but doing 10~20 echo to a log file on the shared file system
>> from many nodes at the same time is expensive, which I think is the main
>> bottleneck in the current wrapper script.  Once Zhao is done disabling all
>> logging, except for necessary ones, we'll have a better idea of how fast we
>> can go, and if it is necessary to eliminate the mkdir step as well. 
>>     
>
> When I was playing with this around the time of SC, I put in a bunch of 
> progress logging inside the wrapper script. This adds to the amount of 
> logging that the wrapper does, but gives a many stage breakdown of where 
> the wrapper script is spending its time.
>
> Run a bunch of jobs, eg a few thousand, with latest SVN and 
> wrapperlog.always.transfer=true set in swift.properties.
>
> You'll get a <runid>.d directory, with a bunch of .info files. From there 
> I (or you) can graph how each wrapper script spent its time.
>
> Ideally there should be a bunch of steps taking almost no time, then the 
> executable, then another bunch of steps taking almost no time; but doing 
> this should reveal wrong behaviour there.
>
> Poke me when you have that dump directory and I can have a look.
>   
Ideally, we'd want any extra logging outside of the bare minimum to be 
optional, something that could be turned on or off depending on output 
level.  Maybe you or Mihael could work in such an option in the future, 
so we could easily disable all logging in the wrapper script if we need 
to.  In the meantime, we'll hack away to it ourselves :)

We'll try to do some back to back comparison runs, and save the logs, 
and let you know where they are for later debugging.

Thanks,
Ioan

-- 
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
http://dev.globus.org/wiki/Incubator/Falkon
http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page
===================================================
===================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20080307/e7dcf0a2/attachment.html>