[Swift-commit] r2270 - log-processing/libexec
noreply at svn.ci.uchicago.edu
noreply at svn.ci.uchicago.edu
Sat Oct 4 21:02:53 CDT 2008
Author: benc
Date: 2008-10-04 21:02:52 -0500 (Sat, 04 Oct 2008)
New Revision: 2270
Added:
log-processing/libexec/assorted.html.template
log-processing/libexec/execute.html.template
log-processing/libexec/execute2.html.template
log-processing/libexec/falkon.html.template
log-processing/libexec/info.html.template
log-processing/libexec/karajan.html.template
log-processing/libexec/kickstart.html.template
log-processing/libexec/scheduler.html.template
log-processing/libexec/toolbar.html.template
Modified:
log-processing/libexec/index.html.template
log-processing/libexec/log-to-karatasks-transitions
log-processing/libexec/makefile
log-processing/libexec/makefile.webpage
Log:
rearrange generated web page into a multipage format, hopefully to make it easier to navigate and provide more structure into which to put fascinating descriptions of what each graph is about...
Added: log-processing/libexec/assorted.html.template
===================================================================
--- log-processing/libexec/assorted.html.template (rev 0)
+++ log-processing/libexec/assorted.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,38 @@
+<html>
+<head><title>Swift log analysis</title></head>
+
+<body><h1>Swift log analysis</h1>
+
+include(`toolbar.html.template')
+
+<h2>Assorted Swift library routines</h2>
+
+<p>Shared directory initialisation on remote sites:</p>
+<img src="initshareddir.png"/>
+
+<p>Per-job shared directory creation on remote sites:</p>
+<img src="createdirset.png"/>
+<br/>
+<img src="createdirset-total.png"/>
+
+<p>dostagein:</p>
+<img src="dostagein.png"/>
+<br/>
+<img src="dostagein-total.png"/>
+<p>dostagein duration histogram:</p>
+<img src="dostagein-duration-histogram.png" />
+<hr />
+
+<p>dostageout:</p>
+<img src="dostageout.png"/>
+<br/>
+<img src="dostageout-total.png"/>
+<p>dostageout sorted by start time</p>
+<img src="dostageout.sorted-start.png"/>
+
+<p>dostageout duration histogram:</p>
+<img src="dostageout-duration-histogram.png" />
+<hr />
+
+<p>end</p>
+</body>
Added: log-processing/libexec/execute.html.template
===================================================================
--- log-processing/libexec/execute.html.template (rev 0)
+++ log-processing/libexec/execute.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,55 @@
+<html>
+<head><title>Swift log analysis</title></head>
+
+<body><h1>Swift log analysis</h1>
+
+include(`toolbar.html.template')
+
+<!-- (TODO indicate log file name here; indicate report generation
+time here; indicate version of swift that was used - if locally modified
+print a big red warning) -->
+
+<hr />
+<h2>Summary of execution</h2>
+<p>This plot shows the state of executions over time. When a SwiftScript
+program makes a procedure call, an execution will appear on this plot
+in START state. Over time, jobs should end up either in END_SUCCESS or
+END_FAILURE states.
+</p>
+<img src="execute-trails.png" />
+
+<p>Final status totals:</p>
+<pre>
+include(`execute.lastsummary')
+</pre>
+
+<h2>execute</h2>
+<p>Executions
+(<a href="execute.event">text file of events</a>)
+(these are what appear as procedure invocations in SwiftScript
+and are rougly analogous to VDL1 derivations):</p>
+<img src="execute.png" />
+<p>The number of execute tasks in progress at once (for each time t, count the
+number of event lines on the above graph that are drawn at time t):</p>
+<img src="execute-total.png"/>
+
+<p>Execute events ordered by start time</p>
+<img src="execute.sorted-start.png" />
+
+<p>Breakdown of last known status for execute:</p>
+<pre>
+include(`execute.lastsummary')
+</pre>
+
+<p>execute trails:</p>
+<img src="execute-trails.png" />
+
+<hr />
+<h2>general stats</h2>
+<p>Some execution counts:</p>
+<pre>
+include(`execution-counts.txt')
+</pre>
+
+<p>end</p>
+</body>
Added: log-processing/libexec/execute2.html.template
===================================================================
--- log-processing/libexec/execute2.html.template (rev 0)
+++ log-processing/libexec/execute2.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,56 @@
+<html>
+<head><title>Swift log analysis</title></head>
+
+<body><h1>Swift log analysis</h1>
+
+include(`toolbar.html.template')
+
+<hr />
+
+<p>Breakdown of execute2 final statuses by site:
+(<a href="execute2.html">More information about execute2 events</a>)</p>
+
+include(`jobs-sites.table')
+
+<p>Number of execute2 jobs at once - this is basically the number of
+execution attempts that have been submitted to some batch system such
+as GRAM (so are either queued or executing)</p>
+<img src="execute2-total.png"/>
+
+<h2>execute2</h2>
+<p>execute2 tasks (<a href="execute2.event">text file of events</a>) - these are executions of programs. Sometimes (in the
+case of failure) there will be multiple execute2 tasks per execute task:</p>
+<img src="execute2.png" />
+<p>number of execute2 tasks at once:</p>
+<img src="execute2-total.png"/>
+<p>execute2 tasks, coloured by site</p>
+<img src="execute2-coloured.png" />
+<p>Breakdown of last known status for execute2s:</p>
+<pre>
+include(`execute2.lastsummary')
+</pre>
+<p>execute2 trails:</p>
+
+<img src="execute2-trails.png" />
+
+<p>Summary of execute2 transformation names that have been seen
+(eg are enqueued or have been run):</p>
+<pre>
+include(`trname-summary')
+</pre>
+
+<p>per-site execute2 duration statistics</p>
+<pre>
+include(`site-duration.txt')
+</pre>
+
+<p>Summary of execute2s/execute:
+</p>
+<pre>
+|number of executes | number of execute2s in each execute |
+include(`jobs.retrycount.summary')
+</pre>
+
+<hr />
+<p>end</p>
+</body>
Added: log-processing/libexec/falkon.html.template
===================================================================
--- log-processing/libexec/falkon.html.template (rev 0)
+++ log-processing/libexec/falkon.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,20 @@
+<html>
+<head><title>Swift log analysis</title></head>
+
+<body><h1>Swift log analysis</h1>
+
+include(`toolbar.html.template')
+
+<h2>falkon</h2>
+<p>Falkon taskperf log:</p>
+
+<img src="falkon.png"/>
+
+<p>Total jobs that falkon has queued or running:</p>
+<img src="falkon-total.png"/>
+
+<p>Falkon job status trails:</p>
+<img src="falkon-task-status-trails.png"/>
+
+<p>end</p>
+</body>
Modified: log-processing/libexec/index.html.template
===================================================================
--- log-processing/libexec/index.html.template 2008-10-05 01:51:54 UTC (rev 2269)
+++ log-processing/libexec/index.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -3,251 +3,32 @@
<body><h1>Swift log analysis</h1>
-<p>(TODO indicate log file name here; indicate report generation
+include(`toolbar.html.template')
+
+<!-- (TODO indicate log file name here; indicate report generation
time here; indicate version of swift that was used - if locally modified
-print a big red warning)</p>
+print a big red warning) -->
-<hr />
<h2>Summary of execution</h2>
<p>This plot shows the state of executions over time. When a SwiftScript
program makes a procedure call, an execution will appear on this plot
in START state. Over time, jobs should end up either in END_SUCCESS or
-END_FAILURE states. (<a href="execute.html">More information about
+END_FAILURE states. (<a href="execute.html">More detail about
execute events</a>)
</p>
<img src="execute-trails.png" />
-<p>Final status totals:</p>
-<pre>
-include(`execute.lastsummary')
-</pre>
-
-<p>Breakdown of execute2 final statuses by site:
-(<a href="execute2.html">More information about execute2 events</a>)</p>
-
-include(`jobs-sites.table')
-
-
-<p>Number of execute2 jobs at once - this is basically the number of
-execution attempts that have been submitted to some batch system such
-as GRAM (so are either queued or executing)</p>
-<img src="execute2-total.png"/>
-
<p>Number of karajan level job submissions that are 'Active' - this means
that the underlying execution layer (such as GRAM) has reported that the
-job is actually executing</p>
+job is actually executing - this is an approximation of the number
+of cores in use at once (for single threaded application code)
+(<a href="karajan.html">More detail about karajan events</a>)</p>
<img src="karatasks.JOB_SUBMISSION.Active-total.png" />
-<hr />
-<h2>execute</h2>
-<p>Executions
-(<a href="execute.event">text file of events</a>)
-(these are what appear as procedure invocations in SwiftScript
-and are rougly analogous to VDL1 derivations):</p>
-<img src="execute.png" />
-<p>The number of execute tasks in progress at once (for each time t, count the
-number of event lines on the above graph that are drawn at time t):</p>
-<img src="execute-total.png"/>
-
-<p>Execute events ordered by start time</p>
-<img src="execute.sorted-start.png" />
-
-<p>Breakdown of last known status for execute:</p>
-<pre>
-include(`execute.lastsummary')
-</pre>
-
-<p>execute trails:</p>
-<img src="execute-trails.png" />
-
-<hr />
-<h2>execute2</h2>
-<p>execute2 tasks (<a href="execute2.event">text file of events</a>) - these are executions of programs. Sometimes (in the
-case of failure) there will be multiple execute2 tasks per execute task:</p>
-<img src="execute2.png" />
-<p>number of execute2 tasks at once:</p>
-<img src="execute2-total.png"/>
-<p>execute2 tasks, coloured by site</p>
-<img src="execute2-coloured.png" />
-<p>Breakdown of last known status for execute2s:</p>
-<pre>
-include(`execute2.lastsummary')
-</pre>
-<p>execute2 trails:</p>
-
-<img src="execute2-trails.png" />
-
-<p>Summary of execute2 transformation names that have been seen
-(eg are enqueued or have been run):</p>
-<pre>
-include(`trname-summary')
-</pre>
-
-
-<hr />
-<h2>kickstart</h2>
-<p>kickstart records.
-(<a href="kickstart.event">text file of events</a>)
-this shows applications which are actually running on a worker node, executing:</p>
-<img src="kickstart.png" />
-
-<p>kickstart records total - this represents how many worker nodes were
-actually executing applications at any time:</p>
-<img src="kickstart-total.png" />
-
-<p>kickstart records sorted by start time:</p>
-<img src="kickstart.sorted-start.png" />
-
-<p>kickstart stats:
-<pre>
-include(`kickstart.stats')
-</pre>
-</p>
-
-<p>cumulative histogram(?) of kickstart job durations</p>
-
-<img src="kickstart-duration-histogram.png" />
-
-<hr />
-<h2>karajan tasks</h2>
-<p>just karajan task start/end
-(<a href="karatasks.event">text file of events</a>):</p>
-<img src="karatasks.png" />
-
-<p>karajan tasks, coloured by type:</p>
-<img src="karatasks-coloured.png" />
-
-<p>karajan tasks, coloured by type and sorted by start time:</p>
-<img src="karatasks-coloured-sorted.png" />
-
-<p>Total karajan tasks in progress at any time:</p>
-<img src="karatasks-total.png" />
-
-<p>Counts for each task type:</p>
-
-<pre>
-include(`karatasks-type-counts.txt')
-</pre>
-
-<p>Breakdown of last known status for karajan tasks:</p>
-<pre>
-include(`karatasks.lastsummary')
-</pre>
-
-<p>karajan FILE_TRANSFER tasks</p>
-<img src="karatasks.FILE_TRANSFER.sorted-start.png" />
-<br/>
-<img src="karatasks.FILE_TRANSFER-total.png" />
-<br/>
-<pre>
-include(`karatasks.FILE_TRANSFER.stats')
-</pre>
-
-<p>karatasks FILE_TRANSFER trails:</p>
-<img src="karatasks.FILE_TRANSFER-trails.png" />
-
-<p>karajan FILE_OPERATION tasks</p>
-<img src="karatasks.FILE_OPERATION.sorted-start.png" />
-<br/>
-<img src="karatasks.FILE_OPERATION-total.png" />
-<br/>
-<pre>
-include(`karatasks.FILE_OPERATION.stats')
-</pre>
-<p>karatasks FILE_OPERATION trails:</p>
-<img src="karatasks.FILE_OPERATION-trails.png" />
-<p>karajan JOB_SUBMISSION tasks</p>
-<img src="karatasks.JOB_SUBMISSION.sorted-start.png" />
-<br/>
-<img src="karatasks.JOB_SUBMISSION-total.png" />
-<br/>
-<pre>
-include(`karatasks.JOB_SUBMISSION.stats')
-</pre>
-<p>karatasks JOB_SUBMISSION trails:</p>
-<img src="karatasks.JOB_SUBMISSION-trails.png" />
-
-<p>Number of karajan level job submissions that are 'Submitted' - this means
-that the underlying execution layer (such as GRAM) has reported that the
-job is submitted for execution, but not executing (queued).</p>
-<img src="karatasks.JOB_SUBMISSION.Queue-total.png" />
-
-<p>karajan queued JOB_SUBMISSION tasks</p>
-<img src="karatasks.JOB_SUBMISSION.Queue.sorted-start.png" />
-
-<p>karajan queued JOB_SUBMISSION cumulative duration</p>
-<img src="karatasks.JOB_SUBMISSION.Queue-duration-histogram.png" />
-
-<pre>
-include(`karatasks.JOB_SUBMISSION.Queue.stats')
-</pre>
-<hr />
-
-<p>Number of tasks in Active state</p>
-<img src="karatasks.JOB_SUBMISSION.Active-total.png" />
-
-<p>karajan active JOB_SUBMISSION tasks</p>
-<img src="karatasks.JOB_SUBMISSION.Active.sorted-start.png" />
-
-<p>karajan active JOB_SUBMISSION cumulative duration</p>
-<img src="karatasks.JOB_SUBMISSION.Active-duration-histogram.png" />
-
-<pre>
-include(`karatasks.JOB_SUBMISSION.Active.stats')
-</pre>
-<hr />
-
-<hr />
-<h2>falkon</h2>
-<p>Falkon taskperf log:</p>
-
-<img src="falkon.png"/>
-
-<p>Total jobs that falkon has queued or running:</p>
-<img src="falkon-total.png"/>
-
-<p>Falkon job status trails:</p>
-<img src="falkon-task-status-trails.png"/>
-
-<h2>various Swift karajan-level routines</h2>
-
-<p>Shared directory initialisation on remote sites:</p>
-<img src="initshareddir.png"/>
-
-<p>Per-job shared directory creation on remote sites:</p>
-<img src="createdirset.png"/>
-<br/>
-<img src="createdirset-total.png"/>
-
-<p>dostagein:</p>
-<img src="dostagein.png"/>
-<br/>
-<img src="dostagein-total.png"/>
-<p>dostagein duration histogram:</p>
-<img src="dostagein-duration-histogram.png" />
-<hr />
-
-<p>dostageout:</p>
-<img src="dostageout.png"/>
-<br/>
-<img src="dostageout-total.png"/>
-<p>dostageout sorted by start time</p>
-<img src="dostageout.sorted-start.png"/>
-
-<p>dostageout duration histogram:</p>
-<img src="dostageout-duration-histogram.png" />
-<hr />
-
-
<p>Various execution-related total-event plots on one chart:</p>
<img src="execstages.png" />
<hr />
-<h2>everything</h2>
-<p>all events together - this one is rather meaningless:</p>
-<img src="all.png" />
-
-<hr />
<h2>general stats</h2>
<p>Some execution counts:</p>
<pre>
@@ -259,71 +40,9 @@
include(`sites-list.data')
</pre>
-<p>sites/success table (according to execute2 events
-<a href="execute2.event">execute2.event</a>):</p>
-
-include(`jobs-sites.table')
-
-<p>per-site execute2 duration statistics</p>
-<pre>
-include(`site-duration.txt')
-</pre>
-
-<p>Summary of execute2s/execute:
-</p>
-<pre>
-|number of executes | number of execute2s in each execute |
-include(`jobs.retrycount.summary')
-</pre>
-
-<hr />
-<h2>scheduler scores</h2>
-<p>Scheduler host scores:</p>
-<img src="weights.png" />
-
-<p>Scheduler load information:</p>
-<img src="tscores.png" />
-
-<hr />
-<h2>worker.sh info files</h2>
-<p>info file plots:</p>
-<p>info files:</p>
-<img src="info.png" />
-<p>info files sorted by start time:</p>
-<img src="info.sorted-start.png" />
-<p>info trails:</p>
-<img src="info-trails.png" />
-<p>info duration histogram:</p>
-<img src="info-duration-histogram.png" />
-<p>how wrapper.sh is spending its time:</p>
-<img src="info.zeroed-trailsx.png" />
-
-<p>total concurrent wrapper+application execution on worker nodes</p>
-
-<img src="info-total.png" />
-
-<p>info/wrapperlog stats:
-<pre>
-include(`info.stats')
-</pre>
-</p>
-<hr />
-
-
-1. <img src="karatasks.clusters.png" />
-<br />
-2. <img src="karatasks.clusters-total.png" />
-<br />
-3. <img src="karatasks.clusters.sorted-start.png" />
-<br />
-4. <img src="karatasks.clusters-duration-histogram.png" />
-<br />
-5. <img src="cluster-jobcount-histogram.png" />
-<hr />
-
<h2>Workflow</h2>
-<p>The workflow itself
+<p>The whole workflow
(<a href="workflow.event">text file of event(s)</a>)
- there should be just one of them and it should
take the whole span of execution time:</p>
Added: log-processing/libexec/info.html.template
===================================================================
--- log-processing/libexec/info.html.template (rev 0)
+++ log-processing/libexec/info.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,29 @@
+<html>
+<head><title>Swift log analysis</title></head>
+
+<body><h1>Swift log analysis</h1>
+
+include(`toolbar.html.template')
+
+<hr />
+<h2>worker.sh info files</h2>
+<p>info file plots:</p>
+<p>info files:</p>
+<img src="info.png" />
+<p>info files sorted by start time:</p>
+<img src="info.sorted-start.png" />
+<p>info trails:</p>
+<img src="info-trails.png" />
+<p>info duration histogram:</p>
+<img src="info-duration-histogram.png" />
+<p>how wrapper.sh is spending its time:</p>
+<img src="info.zeroed-trailsx.png" />
+
+<p>total concurrent wrapper+application execution on worker nodes</p>
+
+<img src="info-total.png" />
+
+<hr />
+<p>end</p>
+</body>
+</html>
Added: log-processing/libexec/karajan.html.template
===================================================================
--- log-processing/libexec/karajan.html.template (rev 0)
+++ log-processing/libexec/karajan.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,121 @@
+<html>
+<head><title>Swift log analysis</title></head>
+
+<body><h1>Swift log analysis</h1>
+
+include(`toolbar.html.template')
+
+<!-- (TODO indicate log file name here; indicate report generation
+time here; indicate version of swift that was used - if locally modified
+print a big red warning) -->
+
+<hr />
+<p>Number of karajan level job submissions that are 'Active' - this means
+that the underlying execution layer (such as GRAM) has reported that the
+job is actually executing</p>
+<img src="karatasks.JOB_SUBMISSION.Active-total.png" />
+
+<hr />
+<h2>karajan tasks</h2>
+<p>just karajan task start/end
+(<a href="karatasks.event">text file of events</a>):</p>
+<img src="karatasks.png" />
+
+<p>karajan tasks, coloured by type:</p>
+<img src="karatasks-coloured.png" />
+
+<p>karajan tasks, coloured by type and sorted by start time:</p>
+<img src="karatasks-coloured-sorted.png" />
+
+<p>Total karajan tasks in progress at any time:</p>
+<img src="karatasks-total.png" />
+
+<p>Counts for each task type:</p>
+
+<pre>
+include(`karatasks-type-counts.txt')
+</pre>
+
+<p>Breakdown of last known status for karajan tasks:</p>
+<pre>
+include(`karatasks.lastsummary')
+</pre>
+
+<p>karajan FILE_TRANSFER tasks</p>
+<img src="karatasks.FILE_TRANSFER.sorted-start.png" />
+<br/>
+<img src="karatasks.FILE_TRANSFER-total.png" />
+<br/>
+<pre>
+include(`karatasks.FILE_TRANSFER.stats')
+</pre>
+
+<p>karatasks FILE_TRANSFER trails:</p>
+<img src="karatasks.FILE_TRANSFER-trails.png" />
+
+<p>karajan FILE_OPERATION tasks</p>
+<img src="karatasks.FILE_OPERATION.sorted-start.png" />
+<br/>
+<img src="karatasks.FILE_OPERATION-total.png" />
+<br/>
+<pre>
+include(`karatasks.FILE_OPERATION.stats')
+</pre>
+<p>karatasks FILE_OPERATION trails:</p>
+<img src="karatasks.FILE_OPERATION-trails.png" />
+<p>karajan JOB_SUBMISSION tasks</p>
+<img src="karatasks.JOB_SUBMISSION.sorted-start.png" />
+<br/>
+<img src="karatasks.JOB_SUBMISSION-total.png" />
+<br/>
+<pre>
+include(`karatasks.JOB_SUBMISSION.stats')
+</pre>
+<p>karatasks JOB_SUBMISSION trails:</p>
+<img src="karatasks.JOB_SUBMISSION-trails.png" />
+
+<p>Number of karajan level job submissions that are 'Submitted' - this means
+that the underlying execution layer (such as GRAM) has reported that the
+job is submitted for execution, but not executing (queued).</p>
+<img src="karatasks.JOB_SUBMISSION.Queue-total.png" />
+
+<p>karajan queued JOB_SUBMISSION tasks</p>
+<img src="karatasks.JOB_SUBMISSION.Queue.sorted-start.png" />
+
+<p>karajan queued JOB_SUBMISSION cumulative duration</p>
+<img src="karatasks.JOB_SUBMISSION.Queue-duration-histogram.png" />
+
+<pre>
+include(`karatasks.JOB_SUBMISSION.Queue.stats')
+</pre>
+<hr />
+
+<p>Number of tasks in Active state</p>
+<img src="karatasks.JOB_SUBMISSION.Active-total.png" />
+
+<p>karajan active JOB_SUBMISSION tasks</p>
+<img src="karatasks.JOB_SUBMISSION.Active.sorted-start.png" />
+
+<p>karajan active JOB_SUBMISSION cumulative duration</p>
+<img src="karatasks.JOB_SUBMISSION.Active-duration-histogram.png" />
+
+<pre>
+include(`karatasks.JOB_SUBMISSION.Active.stats')
+</pre>
+<hr />
+
+<hr />
+<h2>job clusters</h2>
+1. <img src="karatasks.clusters.png" />
+<br />
+2. <img src="karatasks.clusters-total.png" />
+<br />
+3. <img src="karatasks.clusters.sorted-start.png" />
+<br />
+4. <img src="karatasks.clusters-duration-histogram.png" />
+<br />
+5. <img src="cluster-jobcount-histogram.png" />
+<hr />
+
+<p>end</p>
+</body>
Added: log-processing/libexec/kickstart.html.template
===================================================================
--- log-processing/libexec/kickstart.html.template (rev 0)
+++ log-processing/libexec/kickstart.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,32 @@
+<html>
+<head><title>Swift log analysis</title></head>
+
+<body><h1>Swift log analysis</h1>
+
+include(`toolbar.html.template')
+
+<h2>kickstart</h2>
+<p>kickstart records.
+(<a href="kickstart.event">text file of events</a>)
+this shows applications which are actually running on a worker node, executing:</p>
+<img src="kickstart.png" />
+
+<p>kickstart records total - this represents how many worker nodes were
+actually executing applications at any time:</p>
+<img src="kickstart-total.png" />
+
+<p>kickstart records sorted by start time:</p>
+<img src="kickstart.sorted-start.png" />
+
+<p>kickstart stats:
+<pre>
+include(`kickstart.stats')
+</pre>
+</p>
+
+<p>cumulative histogram(?) of kickstart job durations</p>
+
+<img src="kickstart-duration-histogram.png" />
+
+<p>end</p>
+</body>
Modified: log-processing/libexec/log-to-karatasks-transitions
===================================================================
--- log-processing/libexec/log-to-karatasks-transitions 2008-10-05 01:51:54 UTC (rev 2269)
+++ log-processing/libexec/log-to-karatasks-transitions 2008-10-05 02:02:52 UTC (rev 2270)
@@ -1,7 +1,7 @@
#!/bin/bash
# this will take a log file and produce a set of state transition
-# events for every execute2 entity
+# events for every karajan task entity
# grep filters down to mostly relevant lines first - this is quick
# reduces the more expensive later processing
Modified: log-processing/libexec/makefile
===================================================================
--- log-processing/libexec/makefile 2008-10-05 01:51:54 UTC (rev 2269)
+++ log-processing/libexec/makefile 2008-10-05 02:02:52 UTC (rev 2270)
@@ -16,7 +16,7 @@
cp *.event *.html *.png report-$(SDL)
clean:
- rm -f start-times.data kickstart-times.data start-time.tmp end-time.tmp threads.list tasks.list log *.data *.shifted *.png *.event *.coloured-event *.total *.tmp *.transitions *.last karatasks-type-counts.txt index.html *.lastsummary execstages.plot total.plot colour.plot jobs-sites.table jobs.retrycount.summary kickstart.stats execution-counts.txt site-duration.txt jobs.retrycount sp.plot karatasks.coloured-sorted-event *.cedps *.stats t.inf *.seenstates tmp-* info.* clusterstats trname-summary sites-list.data.nm info-md5sums pse2d-tmp.eip
+ rm -f start-times.data kickstart-times.data start-time.tmp end-time.tmp threads.list tasks.list log *.data *.shifted *.png *.event *.coloured-event *.total *.tmp *.transitions *.last karatasks-type-counts.txt index.html *.lastsummary execstages.plot total.plot colour.plot jobs-sites.table jobs.retrycount.summary kickstart.stats execution-counts.txt site-duration.txt jobs.retrycount sp.plot karatasks.coloured-sorted-event *.cedps *.stats t.inf *.seenstates tmp-* clusterstats trname-summary sites-list.data.nm info-md5sums pse2d-tmp.eip
t.inf: $(LOG)
./compute-t-inf > t.inf < $(LOG)
@@ -60,9 +60,6 @@
execute-start-times.data: log
./extract-execute-start-times > execute-start-times.data
-all.event: events
- cat workflow.event execute.event execute2.event karatasks.event kickstart.event > all.event
-
sites-list.data: execute2.event
cat execute2.event | cut -f 6 -d ' ' | sort | uniq > sites-list.data
Modified: log-processing/libexec/makefile.webpage
===================================================================
--- log-processing/libexec/makefile.webpage 2008-10-05 01:51:54 UTC (rev 2269)
+++ log-processing/libexec/makefile.webpage 2008-10-05 02:02:52 UTC (rev 2270)
@@ -1,5 +1,8 @@
-webpage: pngs index.html
+webpage: pngs htmls
+htmls: index.html execute2.html execute.html karajan.html info.html \
+assorted.html kickstart.html falkon.html scheduler.html
+
webpage.kara: karatasks.png karatasks-coloured.png karatasks-type-counts.txt \
karatasks.lastsummary karatasks-coloured-sorted.png \
karatasks.FILE_TRANSFER.sorted-start.png \
@@ -26,8 +29,6 @@
webpage.weights: weights.png tscores.png
-webpage.all: all.png
-
webpage.info: info.png info.sorted-start.png info-trails.png \
info-duration-histogram.png info.zeroed-trailsx.png info.stats info-total.png
@@ -42,9 +43,12 @@
dostageout.sorted-start.png dostagein-duration-histogram.png \
dostageout-duration-histogram.png execute-trails.png execute2-trails.png
-index.html: index.html.template execution-counts.txt jobs-sites.table site-duration.txt execute2.lastsummary execute.lastsummary jobs.retrycount.summary trname-summary
- m4 index.html.template > index.html
+htmldeps: index.html.template execution-counts.txt jobs-sites.table site-duration.txt execute2.lastsummary execute.lastsummary jobs.retrycount.summary trname-summary
+%.html: htmldeps
+ m4 $@.template > $@
+
+
index.html.kara: karatasks.lastsummary karatasks.FILE_OPERATION.stats karatasks.FILE_TRANSFER.stats karatasks.JOB_SUBMISSION.stats karatasks-type-counts.txt
index.html.kickstart: kickstart.stats
Added: log-processing/libexec/scheduler.html.template
===================================================================
--- log-processing/libexec/scheduler.html.template (rev 0)
+++ log-processing/libexec/scheduler.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,17 @@
+<html>
+<head><title>Swift log analysis</title></head>
+
+<body><h1>Swift log analysis</h1>
+
+include(`toolbar.html.template')
+
+<hr />
+<h2>scheduler scores</h2>
+<p>Scheduler host scores:</p>
+<img src="weights.png" />
+
+<p>Scheduler load information (tscores):</p>
+<img src="tscores.png" />
+
+<p>end</p>
+</body>
Added: log-processing/libexec/toolbar.html.template
===================================================================
--- log-processing/libexec/toolbar.html.template (rev 0)
+++ log-processing/libexec/toolbar.html.template 2008-10-05 02:02:52 UTC (rev 2270)
@@ -0,0 +1,13 @@
+<p>
+<a href="index.html">Summary</a> |
+Channels:
+<a href="execute.html">execute</a> |
+<a href="execute2.html">execute2</a> |
+<a href="karajan.html">karajan</a> |
+<a href="kickstart.html">kickstart</a> |
+<a href="info.html">info</a> |
+<a href="falkon.html">falkon</a> |
+<a href="assorted.html">assorted swift library routines</a>
+| Other:
+<a href="scheduler.html">scheduler</a>
+</p>
More information about the Swift-commit
mailing list