<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
.OK, here is my analysis of the plateaus, from Falkon's point of view.<br>
<br>
Notice the per task execution (green) is about 100 seconds per job,
where the job is some invocation of the wrapper.sh that Swift sent to
Falkon. Things look normal so far. See the 2nd graph for more...<br>
<br>
<img src="cid:part1.02090803.01060807@cs.uchicago.edu" alt=""><br>
This shows that there are 600 workers (600 CPUs), which all get their
work within 10 seconds... then they all churn away until about 100 sec
when jobs start completing, and new ones get dispatched. At around 132
seconds, the wait queue is empty, and some workers start becoming idle
(the red area)... by time 155, the initial 600 jobs that started
between time 0 and 10, have completed, and from 155 to 211, the
remaining 400 jobs all run to completion; they really only start
completing around 190 sec, and all finish by 211. So, the plateau,
that is evident here as well, is really when 400 workers are executing
400 jobs in parallel, and since the jobs are taking around 100 sec each
to complete, the plateau of 50 seconds is completely normal. See more
after the graph...<br>
<img src="cid:part2.09040602.02080101@cs.uchicago.edu" alt=""><br>
<br>
Now the real question is, what is the breakdown of the 100 sec
invocation (108.645 sec on average to be exact), how much is due to
wrapper.sh, and how much is due to the application itself? Mike, can
you comment on this? I assume you are running amiga which should have
0.5 sec jobs, right?<br>
<br>
Ioan<br>
<br>
Ioan Raicu wrote:
<blockquote cite="mid:47E7B27D.6050606@cs.uchicago.edu" type="cite">I
see the plateau, but there are other graphs which seem to go crazy
during those periods, such as <br>
<a class="moz-txt-link-freetext"
href="http://www.ci.uchicago.edu/%7Ebenc/report-amps1-20080323-1935-su38n0k5/karatasks.FILE_TRANSFER-total.png">http://www.ci.uchicago.edu/~benc/report-amps1-20080323-1935-su38n0k5/karatasks.FILE_TRANSFER-total.png</a>
<br>
<a class="moz-txt-link-freetext"
href="http://www.ci.uchicago.edu/%7Ebenc/report-amps1-20080323-1935-su38n0k5/karatasks.FILE_OPERATION-total.png">http://www.ci.uchicago.edu/~benc/report-amps1-20080323-1935-su38n0k5/karatasks.FILE_OPERATION-total.png</a>
<br>
<br>
Looking at the Falkon logs might reveal more about if the plateau was
due to Falkon or not. Where would I find the Falkon logs that
correlate to these graphs? <br>
<br>
Ioan <br>
<br>
Ben Clifford wrote: <br>
<blockquote type="cite">you can get plots for your 1000 job run here:
<br>
<br>
<a class="moz-txt-link-freetext"
href="http://www.ci.uchicago.edu/%7Ebenc/report-amps1-20080323-1935-su38n0k5/">http://www.ci.uchicago.edu/~benc/report-amps1-20080323-1935-su38n0k5/</a>
<br>
<br>
you're hitting the file transfer and file operation limits (that are 20
in your config) once jobs start staging out. <br>
<br>
There's a wierd looking plateu in graph 'number of execute2 tasks at
once:' around 170s .. 200s where no jobs complete for some time. <br>
<br>
Getting the falkon logs and/or the wrapper (.d) logs would be
interesting there. <br>
<br>
these were generated on my laptop with: <br>
<br>
make \ <br>
LOG=/Users/benc/work/everylog/amps1-20080323-1935-su38n0k5.log clean \
<br>
webpage.weights webpage.kara webpage <br>
<br>
using the SVN log-procesisng code. <br>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: <a class="moz-txt-link-abbreviated"
href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a>
Web: <a class="moz-txt-link-freetext"
href="http://www.cs.uchicago.edu/%7Eiraicu">http://www.cs.uchicago.edu/~iraicu</a>
<a class="moz-txt-link-freetext"
href="http://dev.globus.org/wiki/Incubator/Falkon">http://dev.globus.org/wiki/Incubator/Falkon</a>
<a class="moz-txt-link-freetext"
href="http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page">http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page</a>
===================================================
===================================================
</pre>
</body>
</html>