<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
I managed to do it with a series of grep commands.<br>
<br>
I did:<br>
cat dc-6000.log | grep "JOB_SUBMISSION" | grep "TaskImpl" | grep
"Active" > dc-6000-active-completed.txt<br>
cat dc-6000.log | grep "JOB_SUBMISSION" | grep "TaskImpl" | grep
"Completed" >> dc-6000-active-completed.txt<br>
<br>
This new parsed log, was then fed through 3 more programs:<br>
java ParseSwiftLog2 dc-6000-active-completed.txt >
falkon_task_perf.txt<br>
java NormalizeTaskPerf falkon_task_perf.txt >
falkon_task_perf_normalized.txt<br>
java ConvertPerTaskToSummary falkon_task_perf_normalized.txt 1 >
falkon_summary.txt<br>
<br>
and voila, I had the 2 logs Falkon usually generates, the per task log,
and the summary log. I could then run:<br>
java CompareRuns dc-4000/falkon_task_perf.txt
dc-6000/falkon_task_perf.txt 131077<br>
<br>
to get basic statistics on the run and comparison between runs. I
could also run the standard Falkon plots on these logs as well.<br>
<br>
Ioan<br>
<br>
Ben Clifford wrote:
<blockquote
cite="mid:Pine.LNX.4.64.0910290348410.14799@dildano.hawaga.org.uk"
type="cite">
<pre wrap="">you can use swift-plot-log ot do that
swift-plot-log my.log karatasks.transitions
or something like that.
don't reimplement all of that yourself.
On Tue, 27 Oct 2009, Ioan Raicu wrote:
</pre>
<blockquote type="cite">
<pre wrap="">OK, so this looks to be a bit more complex then. Can you suggest a way to
extract the Active and Complete timestamps (along with the task ID)? It seems
that if I grep for JOB_SUBMISSION, the status Active is not found on the same
line, so I can't filter according to those two keywords. Also, the Completed
keyword also comes up with many other hits. Any ideas on how to get the info I
need (with cat, grep, etc) without writing some custom script/program that
actually understands in depth the logging of the Swift log?
Thanks,
Ioan
Mihael Hategan wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Tue, 2009-10-27 at 14:17 -0500, Ioan Raicu wrote:
</pre>
<blockquote type="cite">
<pre wrap="">OK, I converted the logs, and here is what I got.
The thing that bugs me is that things ran faster (per task) at the
larger scale (a bit counter intuitive).
</pre>
</blockquote>
<pre wrap="">Nope. Same workload but more processors.
</pre>
<blockquote type="cite">
<pre wrap=""> Also, the maximum number of concurrent tasks I found was 20K, in both
experiments. This doesn't seem right, as the experiments should have
4K or 6K active tasks at any time.
</pre>
</blockquote>
<pre wrap="">Right. You should see around 4k or 6k max active tasks.
</pre>
<blockquote type="cite">
<pre wrap=""> Could I be looking at the wrong entries from the Swift logs?
I did this to get the data I needed from the Swift logs:
cat dc-4000.log | grep "JOB_START" > dc-4000-start-end.txt
cat dc-4000.log | grep "JOB_END" >> dc-4000-start-end.txt
Is this correct? Or should I be looking for other job events?
</pre>
</blockquote>
<pre wrap="">You should probably be looking for "Task(type=JOB_SUBMISSION...) setting
status to Active" and "Task(type=JOB_SUBMISSION...) setting status to
Completed". Or something like that.
JOB_START and JOB_END are statuses for the Swift execute2 processes.
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384
Evanston, IL 60208-3118
=================================================================
Cel: 1-847-722-0876
Tel: 1-847-491-8163
Email: <a class="moz-txt-link-abbreviated" href="mailto:iraicu@eecs.northwestern.edu">iraicu@eecs.northwestern.edu</a>
Web: <a class="moz-txt-link-freetext" href="http://www.eecs.northwestern.edu/~iraicu/">http://www.eecs.northwestern.edu/~iraicu/</a>
<a class="moz-txt-link-freetext" href="https://wiki.cucis.eecs.northwestern.edu/">https://wiki.cucis.eecs.northwestern.edu/</a>
=================================================================
=================================================================
</pre>
</body>
</html>