[Swift-devel] Notes for log processing

[Michael Wilde]

I wanted to do some basic plots to better understand the performance and bottlenecks of a Swift script. I started with this doc on log processing:

  http://www.ci.uchicago.edu/swift/wwwdev/guides/trunk/userguide/userguide.html#_log_processing

Here's what I had to do to generate a load plot of my Swift run. Hopefully this will help in documenting how to use the new Java plotting configs now, as well as packaging them for easy use with higher level scripts.

- Mike


# Point to the log processing tools

lp=/homes/wilde/swift/src/0.93/cog/modules/swift/dist/swift-svn/libexec/log-processing

# fix normalize-log.pl: FH -> START

# find the start time of the log in decimal unixtime

$ $lp/normalize-log.pl stime amwg_stats-20110922-0541-ii4a5z96.iso >amwg_stats-20110922-0541-ii4a5z96.norm

# convert ISO time to UNIX time in decimal seconds

$ $lp/iso-to-secs <amwg_stats-20110922-0541-ii4a5z96.log >amwg_stats-20110922-0541-ii4a5z96.iso

# Normalize the log to start at time 0.0

$ $lp/normalize-log.pl stime amwg_stats-20110922-0541-ii4a5z96.iso >amwg_stats-20110922-0541-ii4a5z96.norm

This gives, eg:

0 DEBUG Loader arguments: [-config, cf.properties, ...
0.00199985504150391 DEBUG Loader Max heap: 257294336
0.0199999809265137 DEBUG textfiles BEGIN CDM FILE:
0.0199999809265137 DEBUG textfiles END CDM FILE:
0.622999906539917 DEBUG textfiles BEGIN SWIFTSCRIPT:
0.623999834060669 DEBUG textfiles END SWIFTSCRIPT:

# Install the new plotter tools                                                                                                     

$ svn co https://svn.ci.uchicago.edu/svn/vdl2/usertools/plotter                                                                     
 U   plotter
Checked out revision 5151.
fusion$ cd plotter
fusion$ ant
Buildfile: build.xml

compile:
    [javac] Compiling 7 source files to /fusion/gpfs/home/wilde/amwg/run01/plotter/build

jar:
      [jar] Building jar: /fusion/gpfs/home/wilde/amwg/run01/plotter/lib/plotter.jar

BUILD SUCCESSFUL
Total time: 2 seconds


# Edit load.fg (or change your plot files to match it)

# cp $lp/load.cfg . # then edit to contain:

xlabel = time
ylabel = load

shape.amwg_stats-20110922-0541-ii4a5z96.load.data = none
label.amwg_stats-20110922-0541-ii4a5z96.load.data = load

# (was eg: label.load.data = load)

# Generate a Load plot:

$ ./plotter/swift_plotter.zsh -s ./load.cfg load.eps *96.load.data