[Swift-commit] r6237 - in trunk/docs: cookbook userguide
ketan at ci.uchicago.edu
ketan at ci.uchicago.edu
Tue Feb 5 20:30:50 CST 2013
Author: ketan
Date: 2013-02-05 20:30:49 -0600 (Tue, 05 Feb 2013)
New Revision: 6237
Modified:
trunk/docs/cookbook/log_processing
trunk/docs/userguide/log-processing
Log:
log processing
Modified: trunk/docs/cookbook/log_processing
===================================================================
--- trunk/docs/cookbook/log_processing 2013-02-05 15:43:43 UTC (rev 6236)
+++ trunk/docs/cookbook/log_processing 2013-02-06 02:30:49 UTC (rev 6237)
@@ -2,7 +2,7 @@
---------------
To properly generate log plots, you must enable VDL/Karajan logging. Make sure
-log4.properties contains:
+log4j.properties contains:
----
log4j.logger.swift=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
Modified: trunk/docs/userguide/log-processing
===================================================================
--- trunk/docs/userguide/log-processing 2013-02-05 15:43:43 UTC (rev 6236)
+++ trunk/docs/userguide/log-processing 2013-02-06 02:30:49 UTC (rev 6237)
@@ -3,71 +3,24 @@
--------------
To properly generate log plots, you must enable VDL/Karajan logging.
-TODO:How?
+This can be done by putting the following lines in log4j.properties file found in the /etc directory of Swift installation:
-You should check the scripts that you intend to use to determine
-what log lines they require and ensure that you are generating
-those lines via log4j.properties
-
-Make sure log4.properties contains:
--------------------------------------
log4j.logger.swift=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=DEBUG
--------------------------------------
-TODO: Does it work for coasters-based runs only?
-Meaning and interpretation of Swift log messages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+All the executables, zsh and perl scripts mentioned in the following steps are
+available in the libexec/log-processing directory of your Swift installation.
-A Swift log file is typically a text file with the name of the Swift run and
-its timestamp in the filename and an extension ".log". In addition, a ".rlog"
-file is Swift's resume log which is used by Swift when a run is resumed using
-the "-resume" option. The .rlog file is only for Swift's internal purpose and
-not to be interpreted by the user.
-
-Each line in the log file is typically made up of three parts. The first part
-is the timestamp, the second is the type of log message and the third is the
-message itself. The types of log messages follows the java log4j standard types
-of TRACE, DEBUG, INFO, WARN, ERROR and FATAL. This section lists the various
-Swift log messages and explains the meaning and likely interpretation of those
-messages. Please note that the list is not comprehensive at this time. Also
-note that we will ignore the timestamps here.
-
-. _DEBUG Loader arguments: [-sites.file, sites.xml, -config, cf, -tc.file, tc, postproc-gridftp.swift]_
-. _DEBUG Loader Max heap: 5592449024_
-. _DEBUG textfiles BEGIN_
-. _DEBUG Loader kmlversion is_
-. _DEBUG VDL2ExecutionContext Stack dump_
-. _INFO SetFieldValue Set_
-. _INFO get__site STARTCOMPOUND thread=0-8 name=get__site_
-. _INFO vdl:execute START thread=0-8-0 tr=_
-. _INFO GlobalSubmitQueue No global submit throttle set. Using default (1024)_
-. _DEBUG vdl:execute2 THREAD_ASSOCIATION jobid=getsite-ymj72ook thread=0-8-0-1 host=localhost replicationGroup=xmj72ook_
-. _DEBUG vdl:execute2 JOB_START jobid=getsite-ymj72ook tr=getsite arguments=[644] tmpdir=postproc-gridftp-20120319-0942-adf1o1u2/jobs/y/getsite-ymj72ook host=localhost_
-. _INFO GridExec TASK_DEFINITION_
-. _WARN RemoteConfiguration Find: http://140.221.8.62:38260_
-. _INFO AbstractStreamKarajanChannel$Multiplexer Multiplexer 0 started_
-. _INFO AbstractStreamKarajanChannel$Multiplexer (0) Scheduling SC-null for addition_
-. _INFO AbstractStreamKarajanChannel Channel configured_
-. _INFO MetaChannel MetaChannel: 651528505[1478354072: {}] -> null.bind -> SC-null_
-. _INFO ReadBuffer Will ask for 1 buffers for a size of 6070_
-. _INFO ThrottleManager O maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
-. _INFO ThrottleManager mem=113.54 MB, heap=482.88 MB, maxHeap=5.21 GB_
-. _INFO ThrottleManager I maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
-. _INFO PerformanceDiagnosticInputStream [MEM] Heap total: 482.88 MB, Heap used: 118.58 MB_
-. _INFO vdl:execute END_SUCCESS thread=0-8-0 tr=getsite_
-. _INFO WeightedHostScoreScheduler CONTACT_SELECTED host=localhost, score=99.854_
-. _
-
-
Log plotting
~~~~~~~~~~~~
Normalize event times in the log to the run start time
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-* Generate the log, assuming the log is titled +swift-run.log+
+* Generate the normalized log, assuming the log is titled +swift-run.log+
------------------------------------------
./normalize-log.pl file.contains.start.time swift-run.log > swift-run.norm
@@ -148,3 +101,48 @@
------------------------------------------
swift_plotter.zsh -s buckets.cfg buckets.eps buckets.data
------------------------------------------
+
+
+Meaning and interpretation of Swift log messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A Swift log file is typically a text file with the name of the Swift run and
+its timestamp in the filename and an extension ".log". In addition, a ".rlog"
+file is Swift's resume log which is used by Swift when a run is resumed using
+the "-resume" option. The .rlog file is only for Swift's internal purpose and
+not to be interpreted by the user.
+
+Each line in the log file is typically made up of three parts. The first part
+is the timestamp, the second is the type of log message and the third is the
+message itself. The types of log messages follows the java log4j standard types
+of TRACE, DEBUG, INFO, WARN, ERROR and FATAL. This section lists the various
+Swift log messages and explains the meaning and likely interpretation of those
+messages. Please note that the list is not comprehensive at this time. Also
+note that we will ignore the timestamps here.
+
+. _DEBUG Loader arguments: [-sites.file, sites.xml, -config, cf, -tc.file, tc, postproc-gridftp.swift]_
+. _DEBUG Loader Max heap: 5592449024_
+. _DEBUG textfiles BEGIN_
+. _DEBUG Loader kmlversion is_
+. _DEBUG VDL2ExecutionContext Stack dump_
+. _INFO SetFieldValue Set_
+. _INFO get__site STARTCOMPOUND thread=0-8 name=get__site_
+. _INFO vdl:execute START thread=0-8-0 tr=_
+. _INFO GlobalSubmitQueue No global submit throttle set. Using default (1024)_
+. _DEBUG vdl:execute2 THREAD_ASSOCIATION jobid=getsite-ymj72ook thread=0-8-0-1 host=localhost replicationGroup=xmj72ook_
+. _DEBUG vdl:execute2 JOB_START jobid=getsite-ymj72ook tr=getsite arguments=[644] tmpdir=postproc-gridftp-20120319-0942-adf1o1u2/jobs/y/getsite-ymj72ook host=localhost_
+. _INFO GridExec TASK_DEFINITION_
+. _WARN RemoteConfiguration Find: http://140.221.8.62:38260_
+. _INFO AbstractStreamKarajanChannel$Multiplexer Multiplexer 0 started_
+. _INFO AbstractStreamKarajanChannel$Multiplexer (0) Scheduling SC-null for addition_
+. _INFO AbstractStreamKarajanChannel Channel configured_
+. _INFO MetaChannel MetaChannel: 651528505[1478354072: {}] -> null.bind -> SC-null_
+. _INFO ReadBuffer Will ask for 1 buffers for a size of 6070_
+. _INFO ThrottleManager O maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
+. _INFO ThrottleManager mem=113.54 MB, heap=482.88 MB, maxHeap=5.21 GB_
+. _INFO ThrottleManager I maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
+. _INFO PerformanceDiagnosticInputStream [MEM] Heap total: 482.88 MB, Heap used: 118.58 MB_
+. _INFO vdl:execute END_SUCCESS thread=0-8-0 tr=getsite_
+. _INFO WeightedHostScoreScheduler CONTACT_SELECTED host=localhost, score=99.854_
+. _
+
More information about the Swift-commit
mailing list