[Swift-commit] r5002 - branches/release-0.93/docs/userguide

[davidk at ci.uchicago.edu]

Author: davidk
Date: 2011-08-24 15:52:12 -0500 (Wed, 24 Aug 2011)
New Revision: 5002

Added:
   branches/release-0.93/docs/userguide/log-processing
Log:
added chapter


Added: branches/release-0.93/docs/userguide/log-processing
===================================================================
--- branches/release-0.93/docs/userguide/log-processing	                        (rev 0)
+++ branches/release-0.93/docs/userguide/log-processing	2011-08-24 20:52:12 UTC (rev 5002)
@@ -0,0 +1,104 @@
+
+Log Processing
+--------------
+
+To properly generate log plots, you must enable VDL/Karajan logging.
+TODO:How?
+
+
+You should check the scripts that you intend to use to determine
+what log lines they require and ensure that you are generating
+those lines via log4j.properties
+
+Make sure log4.properties contains:
+--------------------------------------
+log4j.logger.swift=DEBUG
+log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
+log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=DEBUG
+--------------------------------------
+TODO: Does it work for coasters-based runs only?
+
+Normalize event times in the log to the run start time
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Generate the log, assuming the log is titled +swift-run.log+
+
+------------------------------------------
+./normalize-log.pl file.contains.start.time swift-run.log > swift-run.norm
+------------------------------------------
+TODO:In what format does the start time be in 'file.contains.start.time'
+
+
+Make a basic load plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Normalize the log.
+. Build up a load data file:
++
+------------------------------------------
+./cpu-job-load.pl < swift-run.norm > load.data
+------------------------------------------
+. Plot with the JFreeChart-based plotter in usertools/plotter:
++
+------------------------------------------
+swift_plotter.zsh -s load.cfg load.eps load.data
+------------------------------------------
+Note: Th load.cfg is available from swift/libexec/log-processing/
+
+
+Make a basic job completion plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Normalize the log.
+
+. Build up a completed data file:
++
+------------------------------------------
+./cpu-job-completed.pl < swift-run.norm > completed.data
+------------------------------------------
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
++
+------------------------------------------
+swift_plotter.zsh -s completed.cfg completed.eps completed.data
+------------------------------------------
+
+Make a basic Block allocation plot from Coasters Block log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Normalize the log.
+
+. Build up a block allocation data file:
++
+------------------------------------------
+./block-level.pl < swift-run.norm > blocks.data
+------------------------------------------
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
++
+------------------------------------------
+swift_plotter.zsh -s blocks.{cfg,eps,data}
+------------------------------------------
+
+Make a job runtime distribution plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Normalize the log.
+
+. Build up a job runtime file:
++
+------------------------------------------
+./extract-times.pl < swift-run.norm > times.data
+------------------------------------------
+
+. Put the job runtimes into 1-second buckets:
++
+------------------------------------------
+./ buckets.pl 1 times.data > buckets.data
+------------------------------------------
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
++
+------------------------------------------
+swift_plotter.zsh -s buckets.cfg buckets.eps buckets.data
+------------------------------------------