<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
I think that using WS-GRAM is key here--it has been created, and
extensively tested, explicitly to address these concerns.<br>
<br>
joseph insley wrote:
<blockquote cite="mid:3B95054D-AECE-438E-8F94-2A7547304372@mcs.anl.gov"
type="cite">I was seeing Mike's jobs show up in the queue, and running
on the backend nodes, and the processes I was seeing on tg-grid
appeared to be gram and not some other application, so it would seem
that it was indeed using PBS.
<div><br class="webkit-block-placeholder">
</div>
<div>However, it appears to be using PRE-WS GRAM.... I still had some
of the 'ps | grep kubal' output in my scrollback:</div>
<div><br class="webkit-block-placeholder">
</div>
<div>
<div>insley@tg-grid1:~> ps -ef | grep kubal </div>
<div>kubal 16981 1 0 16:41 ? 00:00:00
globus-job-manager -conf
/soft/prews-gram-4.0.1-r3/etc/globus-job-manager.conf -type pbs -rdn
jobmanager-pbs -machine-type unknown -publish-jobs</div>
<div>kubal 18390 1 0 16:42 ? 00:00:00
globus-job-manager -conf
/soft/prews-gram-4.0.1-r3/etc/globus-job-manager.conf -type pbs -rdn
jobmanager-pbs -machine-type unknown -publish-jobs</div>
<div>kubal 18891 1 0 16:43 ? 00:00:00
globus-job-manager -conf
/soft/prews-gram-4.0.1-r3/etc/globus-job-manager.conf -type pbs -rdn
jobmanager-pbs -machine-type unknown -publish-jobs</div>
<div>kubal 18917 1 0 16:43 ?</div>
<div><br class="webkit-block-placeholder">
</div>
<div>[snip]</div>
<div><br class="webkit-block-placeholder">
</div>
<div>
<div>kubal 28200 25985 0 16:50 ? 00:00:00 /usr/bin/perl
/soft/prews-gram-4.0.1-r3/libexec/globus-job-manager-script.pl -m pbs
-f /tmp/gram_iwEHrc -c poll</div>
<div>kubal 28201 26954 1 16:50 ? 00:00:00 /usr/bin/perl
/soft/prews-gram-4.0.1-r3/libexec/globus-job-manager-script.pl -m pbs
-f /tmp/gram_lQaIPe -c poll</div>
<div>kubal 28202 19438 1 16:50 ? 00:00:00 /usr/bin/perl
/soft/prews-gram-4.0.1-r3/libexec/globus-job-manager-script.pl -m pbs
-f /tmp/gram_SPsdme -c poll</div>
<div><br class="webkit-block-placeholder">
</div>
<div><br class="webkit-block-placeholder">
</div>
<div>
<div>On Jan 29, 2008, at 1:38 PM, Ioan Raicu wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="margin: 0px;">Can someone double check that the jobs
are using PBS (and not FORK) in GRAM?<span class="Apple-converted-space">
</span>If you are using FORK, then the high load is being caused by
the applications running on the GRAM host.<span
class="Apple-converted-space"> </span>If it is PBS, then I don't
know, others might have more insight.</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Ioan</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Ian Foster wrote:</div>
<blockquote type="cite">
<div style="margin: 0px;">Hi,</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">I've CCed Stuart Martin--I'd greatly
appreciate some insights into what is causing this. I assume that you
are using GRAM4 (aka WS-GRAM)?</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Ian.</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Michael Wilde wrote:</div>
<blockquote type="cite">
<div style="margin: 0px;">[ was Re: Swift jobs on UC/ANL TG ]</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Hi. Im at OHare and will be flying
soon.</div>
<div style="margin: 0px;">Ben or Mihael, if you are online, can
you investigate?</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Yes, there are significant throttles
turned on by default, and the system opens those very gradually.</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">MikeK, can you post to the
swift-devel list your swift.properties file, command line options, and
your swift source code?</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Thanks,</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">MikeW</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">On 1/29/08 8:11 AM, Ti Leggett wrote:</div>
<blockquote type="cite">
<div style="margin: 0px;">The default walltime is 15 minutes.
Are you doing fork jobs or pbs jobs? You shouldn't be doing fork jobs
at all. Mike W, I thought there were throttles in place in Swift to
prevent this type of overrun? Mike K, I'll need you to either stop
these types of jobs until Mike W can verify throttling or only submit a
few 10s of jobs at a time.</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">On Jan 28, 2008, at 01/28/08 07:13
PM, Mike Kubal wrote:</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<blockquote type="cite">
<div style="margin: 0px;">Yes, I'm submitting molecular
dynamics simulations</div>
<div style="margin: 0px;">using Swift.</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Is there a default wall-time
limit for jobs on tg-uc?</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">--- joseph insley <<a
moz-do-not-send="true" href="mailto:insley@mcs.anl.gov">insley@mcs.anl.gov</a>>
wrote:</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<blockquote type="cite">
<div style="margin: 0px;">Actually, these numbers are now
escalating...</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">top - 17:18:54 up<span
class="Apple-converted-space"> </span>2:29,<span
class="Apple-converted-space"> </span>1 user,<span
class="Apple-converted-space"> </span>load average:</div>
<div style="margin: 0px;">149.02, 123.63, 91.94</div>
<div style="margin: 0px;">Tasks: 469 total, <span
class="Apple-converted-space"> </span>4 running, 465 sleeping, <span
class="Apple-converted-space"> </span>0</div>
<div style="margin: 0px;">stopped, <span
class="Apple-converted-space"> </span>0 zombie</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">insley@tg-grid1:~> ps -ef |
grep kubal | wc -l</div>
<div style="margin: 0px;"><span
class="Apple-converted-space"> </span>479</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">insley@tg-viz-login1:~> time
globusrun -a -r</div>
<div style="margin: 0px;">tg-grid.uc.teragrid.org</div>
<div style="margin: 0px;">GRAM Authentication test
successful</div>
<div style="margin: 0px;">real<span
class="Apple-converted-space"> </span>0m26.134s</div>
<div style="margin: 0px;">user<span
class="Apple-converted-space"> </span>0m0.090s</div>
<div style="margin: 0px;">sys <span
class="Apple-converted-space"> </span>0m0.010s</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">On Jan 28, 2008, at 5:15 PM,
joseph insley wrote:</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<blockquote type="cite">
<div style="margin: 0px;">Earlier today
tg-grid.uc.teragrid.org (the UC/ANL</div>
</blockquote>
<div style="margin: 0px;">TG GRAM host)</div>
<blockquote type="cite">
<div style="margin: 0px;">became unresponsive and had
to be rebooted.<span class="Apple-converted-space"> </span>I am</div>
</blockquote>
<div style="margin: 0px;">now seeing slow</div>
<blockquote type="cite">
<div style="margin: 0px;">response times from the
Gatekeeper there again.</div>
</blockquote>
<div style="margin: 0px;">Authenticating to</div>
<blockquote type="cite">
<div style="margin: 0px;">the gatekeeper should only
take a second or two,</div>
</blockquote>
<div style="margin: 0px;">but it is</div>
<blockquote type="cite">
<div style="margin: 0px;">periodically taking up to 16
seconds:</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">insley@tg-viz-login1:~>
time globusrun -a -r</div>
</blockquote>
<div style="margin: 0px;">tg-grid.uc.teragrid.org</div>
<blockquote type="cite">
<div style="margin: 0px;">GRAM Authentication test
successful</div>
<div style="margin: 0px;">real<span
class="Apple-converted-space"> </span>0m16.096s</div>
<div style="margin: 0px;">user<span
class="Apple-converted-space"> </span>0m0.060s</div>
<div style="margin: 0px;">sys <span
class="Apple-converted-space"> </span>0m0.020s</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">looking at the load on
tg-grid, it is rather high:</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">top - 16:55:26 up<span
class="Apple-converted-space"> </span>2:06,<span
class="Apple-converted-space"> </span>1 user,<span
class="Apple-converted-space"> </span>load average:</div>
</blockquote>
<div style="margin: 0px;">89.59, 78.69, 62.92</div>
<blockquote type="cite">
<div style="margin: 0px;">Tasks: 398 total,<span
class="Apple-converted-space"> </span>20 running, 378 sleeping, <span
class="Apple-converted-space"> </span>0</div>
</blockquote>
<div style="margin: 0px;">stopped, <span
class="Apple-converted-space"> </span>0 zombie</div>
<blockquote type="cite">
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">And there appear to be a
large number of processes</div>
</blockquote>
<div style="margin: 0px;">owned by kubal:</div>
<blockquote type="cite">
<div style="margin: 0px;">insley@tg-grid1:~> ps -ef
| grep kubal | wc -l</div>
<div style="margin: 0px;"><span
class="Apple-converted-space"> </span>380</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">I assume that Mike is using
swift to do the job</div>
</blockquote>
<div style="margin: 0px;">submission.<span
class="Apple-converted-space"> </span>Is</div>
<blockquote type="cite">
<div style="margin: 0px;">there some throttling of the
rate at which jobs</div>
</blockquote>
<div style="margin: 0px;">are submitted to</div>
<blockquote type="cite">
<div style="margin: 0px;">the gatekeeper that could be
done that would</div>
</blockquote>
<div style="margin: 0px;">lighten this load</div>
<blockquote type="cite">
<div style="margin: 0px;">some?<span
class="Apple-converted-space"> </span>(Or has that already been done
since</div>
</blockquote>
<div style="margin: 0px;">earlier today?)<span
class="Apple-converted-space"> </span>The</div>
<blockquote type="cite">
<div style="margin: 0px;">current response times are
not unacceptable, but</div>
</blockquote>
<div style="margin: 0px;">I'm hoping to</div>
<blockquote type="cite">
<div style="margin: 0px;">avoid having the machine
grind to a halt as it did</div>
</blockquote>
<div style="margin: 0px;">earlier today.</div>
<blockquote type="cite">
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">Thanks,</div>
<div style="margin: 0px;">joe.</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
</blockquote>
<div style="margin: 0px;">===================================================</div>
<blockquote type="cite">
<div style="margin: 0px;">joseph a.</div>
<div style="margin: 0px;">insley</div>
</blockquote>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<blockquote type="cite">
<div style="margin: 0px;"><a moz-do-not-send="true"
href="mailto:insley@mcs.anl.gov">insley@mcs.anl.gov</a></div>
<div style="margin: 0px;">mathematics & computer
science division</div>
</blockquote>
<div style="margin: 0px;">(630) 252-5649</div>
<blockquote type="cite">
<div style="margin: 0px;">argonne national laboratory</div>
</blockquote>
<div style="margin: 0px;"><span
class="Apple-converted-space"> </span>(630)</div>
<blockquote type="cite">
<div style="margin: 0px;">252-5986 (fax)</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
</blockquote>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">===================================================</div>
<div style="margin: 0px;">joseph a. insley</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;"><a moz-do-not-send="true"
href="mailto:insley@mcs.anl.gov">insley@mcs.anl.gov</a></div>
<div style="margin: 0px;">mathematics & computer
science division <span class="Apple-converted-space"> </span>(630)</div>
<div style="margin: 0px;">252-5649</div>
<div style="margin: 0px;">argonne national laboratory</div>
<div style="margin: 0px;"><span
class="Apple-converted-space"> </span>(630)</div>
<div style="margin: 0px;">252-5986 (fax)</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
</blockquote>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;"><span
class="Apple-converted-space"> </span>____________________________________________________________________________________<span
class="Apple-converted-space"> </span></div>
<div style="margin: 0px;">Be a better friend, newshound, and</div>
<div style="margin: 0px;">know-it-all with Yahoo! Mobile.<span
class="Apple-converted-space"> </span>Try it now.<span
class="Apple-converted-space"> </span><a moz-do-not-send="true"
href="http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ">http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ</a></div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
</blockquote>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
</blockquote>
<div style="margin: 0px;">_______________________________________________</div>
<div style="margin: 0px;">Swift-devel mailing list</div>
<div style="margin: 0px;"><a moz-do-not-send="true"
href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a></div>
<div style="margin: 0px;"><a moz-do-not-send="true"
href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a></div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
</blockquote>
<div style="margin: 0px;">_______________________________________________</div>
<div style="margin: 0px;">Swift-devel mailing list</div>
<div style="margin: 0px;"><a moz-do-not-send="true"
href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a></div>
<div style="margin: 0px;"><a moz-do-not-send="true"
href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a></div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
</blockquote>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px;">--<span class="Apple-converted-space"> </span></div>
<div style="margin: 0px;">==================================================</div>
<div style="margin: 0px;">Ioan Raicu</div>
<div style="margin: 0px;">Ph.D. Candidate</div>
<div style="margin: 0px;">==================================================</div>
<div style="margin: 0px;">Distributed Systems Laboratory</div>
<div style="margin: 0px;">Computer Science Department</div>
<div style="margin: 0px;">University of Chicago</div>
<div style="margin: 0px;">1100 E. 58th Street, Ryerson Hall</div>
<div style="margin: 0px;">Chicago, IL 60637</div>
<div style="margin: 0px;">==================================================</div>
<div style="margin: 0px;">Email: <a moz-do-not-send="true"
href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a></div>
<div style="margin: 0px;">Web: <span class="Apple-converted-space">
</span><a moz-do-not-send="true"
href="http://www.cs.uchicago.edu/%7Eiraicu">http://www.cs.uchicago.edu/~iraicu</a></div>
<div style="margin: 0px;"><a moz-do-not-send="true"
href="http://dev.globus.org/wiki/Incubator/Falkon">http://dev.globus.org/wiki/Incubator/Falkon</a></div>
<div style="margin: 0px;"><a moz-do-not-send="true"
href="http://www.ci.uchicago.edu/wiki/bin/view/VDS/DslCS">http://www.ci.uchicago.edu/wiki/bin/view/VDS/DslCS</a></div>
<div style="margin: 0px;">==================================================</div>
<div style="margin: 0px;">==================================================</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
<div style="margin: 0px; min-height: 14px;"><br>
</div>
</blockquote>
</div>
<br>
<div> <span class="Apple-style-span"
style="border-collapse: separate; border-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate; border-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate; border-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate; border-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px;">
<p style="margin: 0px;"><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal;"
face="Helvetica" size="3">===================================================</font></p>
<p style="margin: 0px;"><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal;"
face="Helvetica" size="3">joseph a. insley
<a moz-do-not-send="true"
href="mailto:insley@mcs.anl.gov">insley@mcs.anl.gov</a></font></p>
<p style="margin: 0px;"><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal;"
face="Helvetica" size="3">mathematics & computer science division
(630) 252-5649</font></p>
<p style="margin: 0px;"><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal;"
face="Helvetica" size="3">argonne national laboratory
(630) 252-5986 (fax)</font></p>
</span></span></span><br class="Apple-interchange-newline">
</span> </div>
<br>
</div>
</div>
</blockquote>
</body>
</html>