[Swift-devel] log-processing

[Michael Wilde]

On 12/10/08 9:41 AM, Ben Clifford wrote:
> On Wed, 10 Dec 2008, Michael Wilde wrote:
> 
>> So I think this would be a good time to make a decision on kickstart 
>> data vs wrapper.sh data. I have no preference. I tend to favor data 
>> thats either tabular or can readily be converted to tabular.
> 
> The user community decided long ago to not use kickstart, I think.

ok. Should we remove the documentation for it and eventually phase it of 
of the code, then? Low prio, but it determines where we put our effort 
in enhancing stats.

> The information from both wrapper logs and kickstart is available from the 
> log processing code in a job-per-line tabular form.

ok. Then next step seems to be documenting it further. When I read
http://www.ci.uchicago.edu/swift/guides/log-processing.php
what Tibi is asking for doesnt jump out at me.

Are you referring to:

"webpage.info - details of execute-site wrapper logs. For versions of 
Swift prior to r1700, you will need to stage the *-info logs back to the 
same place as the Swift log manually. AFter r1700, they are staged back 
automatically under the control of the wrapperlog.always.transfer 
property. The IDIR variable must be set to point to the directory 
containg the logs:

make LOG=/path/fmri-20080304-0901-h8h78lnf.log \
      IDIR=/path/130-fmri-20080304-0901-h8h78lnf.d/ clean webpage.info 
webpage "

?

Sounds like some polishing of this may be the next step then.

So sounds like Tibi, you should try whats there, and report back if it 
meets your needs or not.

>> Ben and/or Mihael, I think you should work out an efficient and 
>> effective architecture for this important stats capture mechanism that 
>> will be useful for users and get it into general use.
> 
> An architecture exists. Its the log-processing code. People even use it! 
> Its likely to be in the Swift 0.8 release. There is plenty of scope for 
> improvement there, but it is the existing base.

OK. Sounds good. I (or better you) can readily guess a few of the main 
things a user will want to ask about a run, and provide examples of the 
inputs and outputs in the user guide. Eg:

"For each app run, create a list of:

appname
sitename
starttime
runtime
cputime
memusage (a few vars)
etc"

"For each app, summarize these by site, by app, etc."

- Mike