[Swift-devel] Request: doStage(In|Out) plots colored per site

[Allan Espinosa]

Hi,

I would like for swift-plot-log to generate transfer time statistics per site so
I added the following modifications to the trunk tree

diff --git a/libexec/log-processing/log-to-dostagein-transitions b/libexec/log-
index 16d0547..f4e8707 100755
--- a/libexec/log-processing/log-to-dostagein-transitions
+++ b/libexec/log-processing/log-to-dostagein-transitions
@@ -12,7 +12,7 @@
 grep ' vdl:dostagein ' | iso-to-secs | \
 grep -E '^[^ ]+ +[^ ]+ +vdl:dostagein ' | \
 sed 's/\([^ ]*\) [^ ]* *vdl:dostagein FILE_STAGE_IN_.*$//' | \
-sed 's/\([^ ]*\) INFO  vdl:dostagein \([^ ]*\) jobid=\([^ ]*\) -.*$/\1 \3 \2/'
+sed 's/\([^ ]*\) INFO  vdl:dostagein \([^ ]*\) jobid=\([^ ]*\) host=\([^ ]*\) 
 grep -E '^[^$]'
 
 exit 0
diff --git a/libexec/log-processing/log-to-dostageout-transitions b/libexec/log
index 07e98e0..5653527 100755
--- a/libexec/log-processing/log-to-dostageout-transitions
+++ b/libexec/log-processing/log-to-dostageout-transitions
@@ -12,7 +12,7 @@
 grep ' vdl:dostageout ' | iso-to-secs | \
 grep -E '^[^ ]+ +[^ ]+ +vdl:dostageout ' | \
 sed 's/\([^ ]*\) [^ ]* *vdl:dostageout FILE_STAGE_OUT_.*$//' | \
-sed 's/\([^ ]*\) INFO  vdl:dostageout \([^ ]*\) jobid=\([^ ]*\) -.*$/\1 \3 \2/
+sed 's/\([^ ]*\) INFO  vdl:dostageout \([^ ]*\) jobid=\([^ ]*\) host=\([^ ]*\)
 grep -E '^[^$]'
 
 exit 0
diff --git a/libexec/vdl-int.k b/libexec/vdl-int.k
index 440617c..aa78e7d 100644
--- a/libexec/vdl-int.k
+++ b/libexec/vdl-int.k
@@ -295,7 +295,7 @@ namespace("vdl"
                                doStageinFile(provider=provider, srchost=srchos
                                                srcdir=srcdir, desthost=host, d
                        )                       
-                       log(LOG:INFO, "END jobid={jobid} - Staging in finished"
+                       log(LOG:INFO, "END jobid={jobid} host={host}- Staging i
                )
 
By the looks of the swift-plot-log tree.  Some colour-dostage(in|out) scripts
should be made to parse these.  

I was wondering if there are other solutions to getting site transfer stats
aside from this proposed hack like querying provenance information or doing
site query per jobid to existing *.event logs so I can reused my old runs.

-Allan