[Swift-commit] r2622 - log-processing

[noreply at svn.ci.uchicago.edu]

Author: benc
Date: 2009-02-27 08:38:31 -0600 (Fri, 27 Feb 2009)
New Revision: 2622

Removed:
   log-processing/README
Log:
remove README from here - relevant details move in a different commit into the docs/log-processing document

Deleted: log-processing/README
===================================================================
--- log-processing/README	2009-02-27 11:51:51 UTC (rev 2621)
+++ log-processing/README	2009-02-27 14:38:31 UTC (rev 2622)
@@ -1,151 +0,0 @@
-This is the README for the swift log-processing package.
-
-Some prerequisites
-==================
-gnuplot 4.0
-gnu m4
-gnu textutils
-perl
-
-Basic use
-=========
-
-To generate a webpage:
-
-add <CHECKOUT>/bin to your path
-
-swift-plot-log modelproc-20080918-1644-nyjqiddf.log
-
-You will be left with a directory
-
-report-modelproc-20080918-1644-nyjqiddf/
-
-in which you will find
-
-report-modelproc-20080918-1644-nyjqiddf/index.html
-
-Open that with your favourite web browser.
-
-
-
-Other Commandline Tools
-=======================
-
-  ./execute2-status-from-log [logfile]
-     lists every (execute2) job and its final status
-
-  ./execute2-summary-from-log [logfile]
-     lists the counts of each final job status in log
-
-There are various event streams, identified by a base name:
-  execute   - an internal procedure in swift
-  execute2  - "
-  dostagein - "
-  dostageout - "
-
-  kickstart - kickstart records (not available as transitions)
-  falkon - complete falkon job executions (not available as transitions)
-
-  karatasks - karajan level tasks, available as transitions
-    (there are also four substreams karatasks.FILE_OPERATION, 
-     karatasks.FILE_TRANSFER and karatasks.JOB_SUBMISSION available
-     as events but not transitions)
-
-  workflow - the workflow execution itself (not available as transitions)
-
-Streams are generated from their source log files either as .transitions
-files or .event files.
-
-If a .transitions file is available, make knows how to convert that into
-an .event file: type  make foo.event
-
-Make knows how to make various plots. For example:
-
- make foo.png   will plot the foo stream
- make foo-total.png  will plot how many foos are in progress at any time
- make foo.sorted-start.png  will plot foos (like foo.png) ordered by start time
-
-Make can also make some statistics:
-
- make foo.stats will output a text file foo.stats
-
-CEPDS
-=====
-
-For any event stream that is available in .transition form,
-   make foo.cedps
-will produces a CEDPS-logging form of those transitions
-
-.event files
-============
-
-need an event file format of one event per line, with that line
-containing start time and duration and other useful data.
-
-col1 = start, col2 = duration, col3 onwards = event specific data - for
-some utilities for now should be column based, but later will maybe
-move to attribute based.
-
-between col 1 and col 2 exactly one space
-between col 2 and col 3 exactly one space
-
-start time is in seconds since unix epoch. start time should *not* be
-normalised to start of workflow
-
-event files should not (for now) be assumed to be in order
-
-different event streams can be stored in different files. each event
-stream should use the extension  .event
-
-
-.coloured-event files
-=====================
-third column is a colour index
-first two columns as per .event (thus a coloured-event is a specific
-form of .event)
-
-
-t=+inf processing
-===================
-
-This is not implemented yet.
-
-For events, the event plots (previously) reflected the time between the
-first and last transitions for an event. In the case of an event being
-'in-progress' at the end of the known log info, this is misleading when
-performing eg. total-events-in-progress graphs - at t=+inf, these events
-should still be in-progress
-
-What is needed is that these events get something like a pseudo-transition
-at t=+inf labelled as "INPROGRESS-AS-FOUND"; with t=+inf being some
-appropriate value (which can perhaps only be determined after all other
-events have been processed such that we can make +inf = maxtime + duration*0.1)
-
-For each channel, make tables of states. (we maybe need this for cdepds).
-On each of these, label events booleanwise with 'in progress' or 'completed'
-so 'WHATEVER_STARTED' is an in-progress event, whilst 'JOB_FAILURE' is not
-an in-progress event.
-
-Then for each event that is in-progress, relabel each event whose final state
-is an inprogress state as 'INPROGRESS-AS-FOUND' with the end time being
-t=+inf
-
-
-for first approx, maxtime is computed by last line of swift logfile
-
-
---- these are the events:
-workflow
-  execute
-    execute2
-      dostagein
-        karatasks (fileops and filetrans)
-      clustering (optional)
-        karatasks (execution)
-          cluster-log (optional)
-            wrapper log (optional)
-              kickstart log
-      dostageout
-        karatasks (fileops and filetrans)
-
----