[Swift-devel] Swift and BGP plots

[Ioan Raicu]

I managed to do it with a series of grep commands.

I did:
cat dc-6000.log | grep "JOB_SUBMISSION" | grep "TaskImpl" | grep 
"Active" > dc-6000-active-completed.txt
cat dc-6000.log | grep "JOB_SUBMISSION" | grep "TaskImpl" | grep 
"Completed" >> dc-6000-active-completed.txt

This new parsed log, was then fed through 3 more programs:
java ParseSwiftLog2 dc-6000-active-completed.txt > falkon_task_perf.txt
java NormalizeTaskPerf falkon_task_perf.txt > 
falkon_task_perf_normalized.txt
java ConvertPerTaskToSummary falkon_task_perf_normalized.txt 1 > 
falkon_summary.txt

and voila, I had the 2 logs Falkon usually generates, the per task log, 
and the summary log. I could then run:
java CompareRuns dc-4000/falkon_task_perf.txt 
dc-6000/falkon_task_perf.txt 131077

to get basic statistics on the run and comparison between runs.  I could 
also run the standard Falkon plots on these logs as well.

Ioan

Ben Clifford wrote:
> you can use swift-plot-log ot do that
>
> swift-plot-log my.log karatasks.transitions
>
> or something like that.
>
> don't reimplement all of that yourself.
>
> On Tue, 27 Oct 2009, Ioan Raicu wrote:
>
>   
>> OK, so this looks to be a bit more complex then. Can you suggest a way to
>> extract the Active and Complete timestamps (along with the task ID)? It seems
>> that if I grep for JOB_SUBMISSION, the status Active is not found on the same
>> line, so I can't filter according to those two keywords. Also, the Completed
>> keyword also comes up with many other hits. Any ideas on how to get the info I
>> need (with cat, grep, etc) without writing some custom script/program that
>> actually understands in depth the logging of the Swift log?
>>
>> Thanks,
>> Ioan
>>
>> Mihael Hategan wrote:
>>     
>>> On Tue, 2009-10-27 at 14:17 -0500, Ioan Raicu wrote:
>>>   
>>>       
>>>> OK, I converted the logs, and here is what I got.
>>>>
>>>> The thing that bugs me is that things ran faster (per task) at the
>>>> larger scale (a bit counter intuitive).
>>>>     
>>>>         
>>> Nope. Same workload but more processors. 
>>>   
>>>       
>>>>  Also, the maximum number of concurrent tasks I found was 20K, in both
>>>> experiments. This doesn't seem right, as the experiments should have
>>>> 4K or 6K active tasks at any time.
>>>>     
>>>>         
>>> Right. You should see around 4k or 6k max active tasks.
>>>
>>>   
>>>       
>>>>  Could I be looking at the wrong entries from the Swift logs?
>>>>
>>>> I did this to get the data I needed from the Swift logs:
>>>> cat dc-4000.log | grep "JOB_START" > dc-4000-start-end.txt
>>>> cat dc-4000.log | grep "JOB_END" >> dc-4000-start-end.txt
>>>>
>>>> Is this correct? Or should I be looking for other job events?
>>>>     
>>>>         
>>> You should probably be looking for "Task(type=JOB_SUBMISSION...) setting
>>> status to Active" and "Task(type=JOB_SUBMISSION...) setting status to
>>> Completed". Or something like that.
>>>
>>> JOB_START and JOB_END are statuses for the Swift execute2 processes.
>>>
>>>
>>>   
>>>       
>>     
>
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20091028/ba86c99d/attachment.html>