[Swift-devel] Re: Swift jobs on UC/ANL TG

[joseph insley]

Actually, these numbers are now escalating...

top - 17:18:54 up  2:29,  1 user,  load average: 149.02, 123.63, 91.94
Tasks: 469 total,   4 running, 465 sleeping,   0 stopped,   0 zombie

insley at tg-grid1:~> ps -ef | grep kubal | wc -l
     479

insley at tg-viz-login1:~> time globusrun -a -r tg-grid.uc.teragrid.org
GRAM Authentication test successful
real    0m26.134s
user    0m0.090s
sys     0m0.010s


On Jan 28, 2008, at 5:15 PM, joseph insley wrote:

> Earlier today tg-grid.uc.teragrid.org (the UC/ANL TG GRAM host)  
> became unresponsive and had to be rebooted.  I am now seeing slow  
> response times from the Gatekeeper there again.  Authenticating to  
> the gatekeeper should only take a second or two, but it is  
> periodically taking up to 16 seconds:
>
> insley at tg-viz-login1:~> time globusrun -a -r tg-grid.uc.teragrid.org
> GRAM Authentication test successful
> real    0m16.096s
> user    0m0.060s
> sys     0m0.020s
>
> looking at the load on tg-grid, it is rather high:
>
> top - 16:55:26 up  2:06,  1 user,  load average: 89.59, 78.69, 62.92
> Tasks: 398 total,  20 running, 378 sleeping,   0 stopped,   0 zombie
>
> And there appear to be a large number of processes owned by kubal:
> insley at tg-grid1:~> ps -ef | grep kubal | wc -l
>     380
>
> I assume that Mike is using swift to do the job submission.  Is  
> there some throttling of the rate at which jobs are submitted to  
> the gatekeeper that could be done that would lighten this load  
> some?  (Or has that already been done since earlier today?)  The  
> current response times are not unacceptable, but I'm hoping to  
> avoid having the machine grind to a halt as it did earlier today.
>
> Thanks,
> joe.
>
>
> ===================================================
> joseph a.  
> insley                                                       
> insley at mcs.anl.gov
> mathematics & computer science division       (630) 252-5649
> argonne national laboratory                               (630)  
> 252-5986 (fax)
>
>

===================================================
joseph a. insley                                                       
insley at mcs.anl.gov
mathematics & computer science division       (630) 252-5649
argonne national laboratory                               (630)  
252-5986 (fax)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20080128/05b26707/attachment.html>