[Swift-devel] Re: plotting using gkrellm

[Ioan Raicu]

BTW, my scripts currently work for a single service.  Running in a 
distributed mode as we do on the real BG/P where each service has their 
own separate logs, will require some aggregation before we can plot the 
logs with gkrellm.  The biggest issue will be how often can be do this 
aggregation, when the logs are spread out over potentially 640 
directories and 640 files on GPFS, that might be under heavy load? My 
guess, is that under light GPFS load and small scale runs, perhaps once 
every second is OK, but as we scale up, and GPFS load will increase, I 
doubt we'll be able to aggregate the logs more often than once every 
minute or so.  I am thinking that for the large scale tests, if we want 
relatively real time feedback (say every second), we can't rely on these 
hundreds of log files... and we'll need a more costom monitoring 
solution that say, transmits the needed log info via TCP to an 
aggregator service every second.  This way, as long as the aggregator 
service can handle the 640 clients with 1 message per second from each, 
then we should be ok. I think sending the needed log info via TCP 
messages will scale better than to append to files, and use some 
external program to parse many files.  In the end, we'll just have to 
try what we have now, to see how well it works, and then try 
alternatives as we see the current techniques not working.

Ioan

Ioan Raicu wrote:
> Hi all,
> I have been thinking about how to plot performance data from Falkon at 
> runtime while experiments are running.  Initially, I tried ploticus 
> and a simple web server, which would regenerate plots every 60 
> seconds, but it was a fragile solution, which didn't always work well. 
> I have always liked how the gkrellm monitor looked and performed, and 
> started thinking about what API it might have. After 5 min of 
> searching through google, I found a plug-in for it that allows you to 
> read an arbitrary text file and plot numerical values by gkrellm :)  
> After a few hours with fiddling with the format of the log files to be 
> read by gkrellm, and the appearance of gkrellm, here is what I have:
> http://people.cs.uchicago.edu/~iraicu/projects/Falkon/plots/Falkon-1M-gkrellm.jpg 
>
> where running 1M tasks, of 60 seconds each, on an emulated BG/P with 
> 160K CPUs. The graph represents about 8 minutes of real time, in which 
> the experiment started from scratch, and completed.
>
> Since it was relatively little effort, and the results are quite nice 
> (especially for demos, I am thinking of SC), I think it would be nice 
> if Swift tried to export some of its log info in the right format to 
> be able to view the data in gkrellm in real time as the experiment 
> progresses.
>
> Here is an example of the log format, which takes the first value of 
> the last line in the falkon-log.txt file, and creates a new file 
> gkrellm-log.txt with the right formated log, ready for gkrellm to read 
> with the fchart plugin 
> (http://lasr.cs.ucla.edu/geoff/gkrellm-fchart.html):
>        tail -n 1 falkon-log.txt | awk '
>        {
>        printf($1 "\n");
>        printf($1 "\n");
>        printf("NORMAL\n");
>        printf($1 "\n");
>        printf("!!TOOLTIP!!\n");
>        printf("!!EOF!!\n");
>        }' > gkrellm-log.tmp
>        cp gkrellm-log.tmp gkrellm-log.txt
>
> I simply have this code in a while loop that goes on forever, and 
> sleeps 1 second between iterations.
>
> Cheers,
> Ioan
>

-- 
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
http://dev.globus.org/wiki/Incubator/Falkon
http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page
===================================================
===================================================