[Swift-commit] r4406 - in www/cookbook: . figures

Mon Apr 25 12:46:59 CDT 2011

Author: ketan
Date: 2011-04-25 12:46:59 -0500 (Mon, 25 Apr 2011)
New Revision: 4406

Modified:
   www/cookbook/cookbook-asciidoc.html
   www/cookbook/cookbook-asciidoc.txt
   www/cookbook/figures/coaster_setup.png
Log:
 

Modified: www/cookbook/cookbook-asciidoc.html
===================================================================

--- www/cookbook/cookbook-asciidoc.html	2011-04-25 03:43:39 UTC (rev 4405)
+++ www/cookbook/cookbook-asciidoc.html	2011-04-25 17:46:59 UTC (rev 4406)
@@ -1100,43 +1100,25 @@
 Its a very interesting question, and a hard but critically important one to
 answer to make usage simpler.</tt></pre>
 </div></div>
-<div class="paragraph"><p>For Advanced Users or How do I fine tune coasters to target specific execution
-environment</p></div>
-<div class="listingblock">
-<div class="content">
-<pre><tt>//Coasters for advanced users. Getting your hands dirty.
-
-. One of the main reason that one would initially deviate from coaster defaults
+<div class="sect2">
+<h3 id="_for_advanced_users_or_how_do_i_fine_tune_coasters_to_target_specific_execution">4.1. For Advanced Users or How do I fine tune coasters to target specific execution</h3>
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+One of the main reason that one would initially deviate from coaster defaults
 into more complex pool entries is to force jobs to fit into some site-imposed
 constraint. For instance a typical submission to the experimental queue requires a user to
 request upto 3 nodes for under 1 hour. This setup could be achieved with a
 careful tuning of coaters parameters.
-
-//How to run Swift under different Coasters configurations
-//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-//
-//Manual
-//^^^^^^^
-//Todo
-//
-//Passive
-//^^^^^^^^
-//Todo
-//
-//Persistent
-//^^^^^^^^^^^
-//Todo
-//
-//Summary of Differences Between different Coaster types
-//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-//A tabular representations of highlights of different coaster setups
-//
-//
-//Data Management
-//
-
-Debugging Swift</tt></pre>
-</div></div>
+</p>
+</li>
+</ol></div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_debugging_swift">5. Debugging Swift</h2>
+<div class="sectionbody">
 <div class="paragraph"><p>Swift errors are logged in several places:</p></div>
 <div class="olist arabic"><ol class="arabic">
 <li>
@@ -1172,8 +1154,100 @@
 </p>
 </li>
 </ol></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_log_processing">6. Log Processing</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>To properly generate log plots, you must enable VDL/Karajan logging.</p></div>
+<div class="paragraph"><p>Make sure log4.properties contains:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>log4j.logger.swift=DEBUG</tt></pre>
+</div></div>
 <div class="sect2">
-<h3 id="_problem_reporting">4.1. Problem Reporting</h3>
+<h3 id="_make_a_basic_load_plot_from_coasters_cpu_log_lines">6.1. Make a basic load plot from Coasters Cpu log lines</h3>
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+Generate the log (may set log4j.logger.swift=INFO for this one)
+(assuming the log is titled swift-run.log)
+</p>
+</li>
+<li>
+<p>
+Convert the log times to Unix time
+./iso-to-secs < swift-run.log > swift-run.time
+</p>
+</li>
+<li>
+<p>
+Make the start time file (this contains the earliest timestamp)
+make LOG=swift-run.log start-time.tmp
+or
+extract-start-time swift-run.log > start-time.tmp
+</p>
+</li>
+<li>
+<p>
+Normalize the transition times
+./normalise-event-start-time < swift-run.time > swift-run.norm
+</p>
+</li>
+<li>
+<p>
+Build up a load data file:
+./cpu-job-load.pl < swift-run.norm > load.data
+</p>
+</li>
+<li>
+<p>
+Plot with the JFreeChart-based plotter in usertools/plotter:
+lines.zsh load.cfg load.eps load.data
+</p>
+</li>
+</ol></div>
+<div class="sect3">
+<h4 id="_make_a_basic_job_completion_plot_from_coasters_cpu_log_lines">6.1.1. Make a basic job completion plot from Coasters Cpu log lines</h4>
+<div class="paragraph"><p>Same as above, but:</p></div>
+<div class="olist arabic"><ol class="arabic" start="5">
+<li>
+<p>
+Build up a completed data file:
+</p>
+</li>
+</ol></div>
+<div class="olist arabic"><div class="title">/cpu-job-completed.pl < swift-run.norm > completed.data</div><ol class="arabic">
+<li>
+<p>
+Plot with the JFreeChart-based plotter in usertools/plotter:
+</p>
+</li>
+</ol></div>
+<div class="paragraph"><p>lines.zsh completed.cfg completed.eps completed.data</p></div>
+</div>
+<div class="sect3">
+<h4 id="_make_a_basic_block_allocation_plot_from_coasters_block_log_lines">6.1.2. Make a basic Block allocation plot from Coasters Block log lines</h4>
+<div class="paragraph"><p>Same as above, but:</p></div>
+<div class="olist arabic"><ol class="arabic" start="5">
+<li>
+<p>
+Build up a block allocation data file:
+</p>
+</li>
+</ol></div>
+<div class="olist arabic"><div class="title">/block-level.pl < swift-run.norm > blocks.data</div><ol class="arabic">
+<li>
+<p>
+Plot with the JFreeChart-based plotter in usertools/plotter:
+lines.zsh blocks.{cfg,eps,data}
+</p>
+</li>
+</ol></div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_problem_reporting">6.2. Problem Reporting</h3>
 <div class="paragraph"><p>When reporting problems to <a href="mailto:swift-user at ci.uchicago.edu">swift-user at ci.uchicago.edu</a>, please attach the
 following files and information:</p></div>
 <div class="olist arabic"><ol class="arabic">
@@ -1237,7 +1311,7 @@
 <div id="footer">
 <div id="footer-text">
 Version 0.92<br />
-Last updated 2011-04-23 20:51:22 CDT
+Last updated 2011-04-25 12:43:22 CDT
 </div>
 </div>
 </body>

Modified: www/cookbook/cookbook-asciidoc.txt
===================================================================
--- www/cookbook/cookbook-asciidoc.txt	2011-04-25 03:43:39 UTC (rev 4405)
+++ www/cookbook/cookbook-asciidoc.txt	2011-04-25 17:46:59 UTC (rev 4406)
@@ -540,9 +540,10 @@
 //
 //
 //
+
 For Advanced Users or How do I fine tune coasters to target specific execution
-environment
-~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 //Coasters for advanced users. Getting your hands dirty.
 
 . One of the main reason that one would initially deviate from coaster defaults
@@ -575,7 +576,7 @@
 //
 
 Debugging Swift
-~~~~~~~~~~~~~~~~
+---------------
 Swift errors are logged in several places:
 
 . All text from standard output and standard error produced by running the
@@ -591,6 +592,68 @@
 Cobalt)
 . $HOME/.globus/??? on remote systems that you access via Globus 
 
+
+Log Processing
+---------------
+
+To properly generate log plots, you must enable VDL/Karajan logging.
+
+Make sure log4.properties contains:
+--------------------------------------
+log4j.logger.swift=DEBUG
+--------------------------------------
+
+Make a basic load plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Generate the log (may set log4j.logger.swift=INFO for this one)
+(assuming the log is titled swift-run.log)
+
+. Convert the log times to Unix time
+./iso-to-secs < swift-run.log > swift-run.time
+
+. Make the start time file (this contains the earliest timestamp)
+make LOG=swift-run.log start-time.tmp
+or 
+extract-start-time swift-run.log > start-time.tmp
+
+. Normalize the transition times
+./normalise-event-start-time < swift-run.time > swift-run.norm
+
+. Build up a load data file:
+./cpu-job-load.pl < swift-run.norm > load.data
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
+lines.zsh load.cfg load.eps load.data
+
+Make a basic job completion plot from Coasters Cpu log lines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Same as above, but:
+
+[start=5]
+. Build up a completed data file:
+
+./cpu-job-completed.pl < swift-run.norm > completed.data
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
+
+lines.zsh completed.cfg completed.eps completed.data
+
+Make a basic Block allocation plot from Coasters Block log lines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Same as above, but:
+
+[start=5]
+. Build up a block allocation data file:
+
+./block-level.pl < swift-run.norm > blocks.data
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
+lines.zsh blocks.{cfg,eps,data}
+
+
 Problem Reporting
 ~~~~~~~~~~~~~~~~~
 When reporting problems to swift-user at ci.uchicago.edu, please attach the

Modified: www/cookbook/figures/coaster_setup.png
===================================================================
(Binary files differ)


