<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
<br>
Mihael Hategan wrote:
<blockquote cite="mid:1213925457.31071.7.camel@localhost" type="cite">
<pre wrap="">There's probably a misunderstanding. Mike seemed to suggest that, when
using BG/P, there should be multiple services in order to distribute
load. </pre>
</blockquote>
Yes, he was correct.<br>
<blockquote cite="mid:1213925457.31071.7.camel@localhost" type="cite">
<pre wrap="">That I think is a problem. </pre>
</blockquote>
I don't follow. If your goal is to just show that it works at small
scales (100s, maybe 1000s of CPUs), you don't need this, but if you
want to have any chance of scaling to 160K CPUs, I don't think you'll
have many options :(<br>
<blockquote cite="mid:1213925457.31071.7.camel@localhost" type="cite">
<pre wrap="">But I also now think he was referring
to the case in which multiple clusters are used, in which case what you
say applies. </pre>
</blockquote>
You can call them whatever you want. It is 1 machine, which is
composed of 640 P-SETs, each P-SET having an I/O node (i.e. think of a
login node from a cluster), and 64 compute nodes (4 CPU cores each
node) on a private network behind each I/O node. So, if you want to
think of the 160K CPU BG/P as a collection of 640 clusters, you can,
but its really 1 big machine.<br>
<br>
The trouble with not using the 640 I/O nodes is that 1 (or a few, up to
10) login nodes has to manage 160K CPUs. If you use the I/O nodes as
well, then 1 (or up to 10) login nodes can manage 640 I/O nodes, which
in turn will each manage 256 CPUs, a break down that is certainly more
manageable. We have had trouble running Falkon reliably at more than
10K~20K CPUs using a single Falkon service (i.e. running on 1 login
node), but when we turned to the hierarchical solution, we have gotten
up to 64K CPUs and it worked without any sign of stress or problems.
BTW, the trouble we had when managing all CPUs from a single service
was probably due to the fact that we were using persistent sockets, and
using select to manage 10K+ active sockets. We have an option to run
without persistent sockets which will scale better, but we haven't
tested this on the BG/P yet as it involves Java on the compute nodes
(which we heard works, but we haven't tried it yet).<br>
<blockquote cite="mid:1213925457.31071.7.camel@localhost" type="cite">
<pre wrap="">We've pretty much discussed this, and (1) is what we would
eventually want to achieve with Swift + Coasters.
</pre>
</blockquote>
Right, I agree that option #1 is the desired goal.<br>
<br>
Ioan<br>
<blockquote cite="mid:1213925457.31071.7.camel@localhost" type="cite">
<pre wrap="">On Thu, 2008-06-19 at 19:58 -0500, Ioan Raicu wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I am not sure of what problem you are referring to fix?
The issue with Falkon, is that there are queues at the service. If a
client submits all its jobs to a single service (that only manages 256
CPUs), there could be 639 other services with 160K - 256 CPUs that are
left idle (worst case, which wouldn't happen very often, but could
still happen towards the ends of runs when there isn't enough work to
keep everyone busy). There are only 2 solutions.
1) never queue anything up at the services, only send tasks from the
client to a service when we know there is an available CPU to run that
task; this is the approach we took
2) allow tasks to timeout after some time, and trigger a resubmit of
the same task to another service, and keep doing this until a reply to
that task comes back; this seems that it would introduce unnecessarily
long delays, and cause load imbalances towards the end of runs when
there isn't enough work to keep all busy
In essence, there is no problem to solve here, its just what solution
you take, in such a distributed tree like environment, where you have
1 client, N services, and M workers. N is a value between 1 and 640,
and M could be as high as 160K, with a ratio of 1:256 between N:M.
Ioan
Mihael Hategan wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Thu, 2008-06-19 at 18:56 -0500, Michael Wilde wrote:
</pre>
<blockquote type="cite">
<pre wrap="">What Ioan did in Falkon when he went to the multiple-server architecture
is relevant here: the client load-shares among all the servers,
round-robin, only sending a job to a server when it knows that the
server has a free cpu slot. In this way, no queues build up on the
servers, and it avoids having a job wait in any server's queue when a
free cpu might be available on some other server.
</pre>
</blockquote>
<pre wrap="">If you have O(1) scheduling, this shouldn't be necessary. It's like
i2u2: Don't build a cluster to reduce the odds of triggering a problem.
Fix the problem instead.
_______________________________________________
Swift-devel mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a>
<a class="moz-txt-link-freetext"
href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a>
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
===================================================
Ioan Raicu
Ph.D. Candidate
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: <a class="moz-txt-link-abbreviated"
href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a>
Web: <a class="moz-txt-link-freetext"
href="http://www.cs.uchicago.edu/%7Eiraicu">http://www.cs.uchicago.edu/~iraicu</a>
<a class="moz-txt-link-freetext"
href="http://dev.globus.org/wiki/Incubator/Falkon">http://dev.globus.org/wiki/Incubator/Falkon</a>
<a class="moz-txt-link-freetext"
href="http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page">http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page</a>
===================================================
===================================================
</pre>
</body>
</html>