[Swift-devel] Re: swift-falkon problem... plots to explain plateaus...

[Ioan Raicu]

.OK, here is my analysis of the plateaus, from Falkon's point of view.

Notice the per task execution (green) is about 100 seconds per job, 
where the job is some invocation of the wrapper.sh that Swift sent to 
Falkon.  Things look normal so far.  See the 2nd graph for more...


This shows that there are 600 workers (600 CPUs), which all get their 
work within 10 seconds... then they all churn away until about 100 sec 
when jobs start completing, and new ones get dispatched.  At around 132 
seconds, the wait queue is empty, and some workers start becoming idle 
(the red area)... by time 155, the initial 600 jobs that started between 
time 0 and 10, have completed, and from 155 to 211, the remaining 400 
jobs all run to completion; they really only start completing around 190 
sec, and all finish by 211.  So, the plateau, that is evident here as 
well, is really when 400 workers are executing 400 jobs in parallel, and 
since the jobs are taking around 100 sec each to complete, the plateau 
of 50 seconds is completely normal.  See more after the graph...


Now the real question is, what is the breakdown of the 100 sec 
invocation (108.645 sec on average to be exact), how much is due to 
wrapper.sh, and how much is due to the application itself?  Mike, can 
you comment on this?  I assume you are running amiga which should have 
0.5 sec jobs, right?

Ioan

Ioan Raicu wrote:
> I see the plateau, but there are other graphs which seem to go crazy 
> during those periods, such as
> http://www.ci.uchicago.edu/~benc/report-amps1-20080323-1935-su38n0k5/karatasks.FILE_TRANSFER-total.png 
>
> http://www.ci.uchicago.edu/~benc/report-amps1-20080323-1935-su38n0k5/karatasks.FILE_OPERATION-total.png 
>
>
> Looking at the Falkon logs might reveal more about if the plateau was 
> due to Falkon or not.  Where would I find the Falkon logs that 
> correlate to these graphs?
>
> Ioan
>
> Ben Clifford wrote:
>> you can get plots for your 1000 job run here:
>>
>> http://www.ci.uchicago.edu/~benc/report-amps1-20080323-1935-su38n0k5/
>>
>> you're hitting the file transfer and file operation limits (that are 
>> 20 in your config) once jobs start staging out.
>>
>> There's a wierd looking plateu in graph 'number of execute2 tasks at 
>> once:' around 170s .. 200s where no jobs complete for some time.
>>
>> Getting the falkon logs and/or the wrapper (.d) logs would be 
>> interesting there.
>>
>> these were generated on my laptop with:
>>
>> make \
>>  LOG=/Users/benc/work/everylog/amps1-20080323-1935-su38n0k5.log clean \
>>  webpage.weights webpage.kara webpage
>>
>> using the SVN log-procesisng code.
>>   
>

-- 
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
http://dev.globus.org/wiki/Incubator/Falkon
http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page
===================================================
===================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20080324/9cef415c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 38387 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20080324/9cef415c/attachment.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 47424 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20080324/9cef415c/attachment-0001.jpe>