<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Actually, these numbers are now escalating...<div><br class="webkit-block-placeholder"></div><div><div>top - 17:18:54 up 2:29, 1 user, load average: 149.02, 123.63, 91.94</div><div>Tasks: 469 total, 4 running, 465 sleeping, 0 stopped, 0 zombie</div><div><br class="webkit-block-placeholder"></div><div><div>insley@tg-grid1:~> ps -ef | grep kubal | wc -l</div><div> 479</div><div><br class="webkit-block-placeholder"></div></div><div><div>insley@tg-viz-login1:~> time globusrun -a -r tg-grid.uc.teragrid.org </div><div>GRAM Authentication test successful</div><div>real 0m26.134s</div><div>user 0m0.090s</div><div>sys 0m0.010s</div><div><br class="webkit-block-placeholder"></div><div><br class="webkit-block-placeholder"></div></div><div><div>On Jan 28, 2008, at 5:15 PM, joseph insley wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Earlier today tg-grid.uc.teragrid.org (the UC/ANL TG GRAM host) became unresponsive and had to be rebooted. I am now seeing slow response times from the Gatekeeper there again. Authenticating to the gatekeeper should only take a second or two, but it is periodically taking up to 16 seconds:</div><div><br class="webkit-block-placeholder"></div><div><div>insley@tg-viz-login1:~> time globusrun -a -r tg-grid.uc.teragrid.org </div><div>GRAM Authentication test successful</div><div>real 0m16.096s</div><div>user 0m0.060s</div><div>sys 0m0.020s</div><div><br class="webkit-block-placeholder"></div><div>looking at the load on tg-grid, it is rather high:</div></div><div><br class="webkit-block-placeholder"></div><div>top - 16:55:26 up 2:06, 1 user, load average: 89.59, 78.69, 62.92</div><div>Tasks: 398 total, 20 running, 378 sleeping, 0 stopped, 0 zombie</div><div><br class="webkit-block-placeholder"></div><div>And there appear to be a large number of processes owned by kubal:</div><div><div>insley@tg-grid1:~> ps -ef | grep kubal | wc -l</div><div> 380</div><div><br class="webkit-block-placeholder"></div><div>I assume that Mike is using swift to do the job submission. Is there some throttling of the rate at which jobs are submitted to the gatekeeper that could be done that would lighten this load some? (Or has that already been done since earlier today?) The current response times are not unacceptable, but I'm hoping to avoid having the machine grind to a halt as it did earlier today.</div><div><br class="webkit-block-placeholder"></div><div>Thanks,</div><div>joe.</div><div><br class="webkit-block-placeholder"></div><br><div> <span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">===================================================</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">joseph a. insley <a href="mailto:insley@mcs.anl.gov">insley@mcs.anl.gov</a></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">mathematics & computer science division (630) 252-5649</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">argonne national laboratory (630) 252-5986 (fax)</font></div></span></span></span><br class="Apple-interchange-newline"></span> </div><br></div></blockquote></div><br><div> <span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">===================================================</font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">joseph a. insley <a href="mailto:insley@mcs.anl.gov">insley@mcs.anl.gov</a></font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">mathematics & computer science division (630) 252-5649</font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">argonne national laboratory (630) 252-5986 (fax)</font></p></span></span></span><br class="Apple-interchange-newline"></span> </div><br></div></body></html>