<div dir="ltr">Great, I think that's a good idea. Most of how it works should be described in the trunk userguide - how to handle run directories, config syntax, search paths for swift.properties, templating, etc.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 4, 2014 at 6:21 PM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@anl.gov" target="_blank">wilde@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
David,<br>
<br>
Mihael suggested making the new config native to Swift, by
converting the perl config code into Java.<br>
<br>
That sounded pretty reasonable, and would make the format you
devised be the sole external specification of all properties.<br>
<br>
We were thinking that this could be done in 0.96. Thoughts?<br>
<br>
- Mike<div><div class="h5"><br>
<br>
<div>On 7/4/14, 6:05 PM, David Kelly wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">This looks great - lots of very useful changes.
I'll volunteer to make the new config work with this sites.xml
format.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Jul 4, 2014 at 3:07 AM, Mihael
Hategan <span dir="ltr"><<a href="mailto:hategan@mcs.anl.gov" target="_blank">hategan@mcs.anl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I committed a fairly large number of changes. Some things
got limited<br>
testing, and some are work in progress. For example, I need
to update<br>
sites files used in tests, but it's getting late here.<br>
Anyway, here's a summary:<br>
<br>
1. New sites.xml format and removal of tc.data. I expect a
bunch of<br>
things to break, but we discussed cleaning both sites.xml
and tc.data<br>
many times and I guess I never got the courage to do it
until now. This<br>
format is not backwards compatible (intentionally). It goes
like this:<br>
<sites><br>
<apps>? <!-- global apps --><br>
<property name="...">...</property>* <!--
global app properties --><br>
<env name="...">...</env>* <!-- global
app envs --><br>
<app name="..." executable="...">*<br>
<property/>*<br>
<env/>*<br>
</app><br>
</apps><br>
<br>
<site name="..."><br>
<execution provider="..."...><br>
<property/>*<br>
</execution><br>
<filesystem>?<br>
<property/>*<br>
</filesystem><br>
<wokdirectory/><br>
<scratch/>?<br>
<apps/>?<br>
<site><br>
</sites><br>
<br>
There are no more namespaces since they were mostly used to
figure out<br>
whether a property was going to the task or the site. I also
changed<br>
"profile" to "property", since "profile" seemed a bit
arcane.<br>
<br>
You should also now be able to use maxParallelTasks and<br>
initialParallelTasks instead of jobThrottle and
initialScore. It<br>
computes the latter automatically.<br>
<br>
2. Coasters now support multiple configurations. So you can
have two<br>
local:local coaster sites with different settings and it
should work.<br>
The way this works is that every configuration (and every
different run)<br>
gets its own job queue, settings, and block allocator, and
they don't<br>
interact with each other. This might be problematic with
things like<br>
slots which is meant to limit the number of jobs globally,
but that's a<br>
minor annoyance.<br>
<br>
3. Passive workers can be launched with a -c(oncurrency)
argument (1 by<br>
default). This will be the jobs per node setting. Each
worker can have a<br>
different number, and that should work as expected. For
passive workers,<br>
the client side jobsPerNode setting will be ignored.<br>
<br>
4. There is a tool to analyze logs now (swift-log-info).
This is simply<br>
an offline version of the http monitor. It can parse logs
and feed the<br>
information to a web browser. It should also be able to
follow logs as<br>
they are produced and feed live information to a browser. It
can also<br>
fake a live log by parsing it slowly instead of all-at-once.<br>
<br>
5. The http monitor got a bit of an update. You can look at
all kinds of<br>
statistics, browse through apps, look at pretty plots of how
apps<br>
behaved, etc. all in a nice dynamically updated ajaxy and
hyperlinked<br>
fashion. It should be cute and useful in one. Won't work
with older<br>
logs. Some pics attached.<br>
<br>
6. There are now <a href="http://worker.pl" target="_blank">worker.pl</a>
"probes" for CPU usage, disk usage, and some<br>
I/O stats. If you click on a link to a worker in the http
monitor,<br>
you'll get nice graphs of these things. The probes are
currently<br>
hardcoded to run every 60 seconds (although in the pictures
below they<br>
ran at 1/s).<br>
<span><font color="#888888"><br>
Mihael<br>
</font></span><br>
_______________________________________________<br>
Swift-devel mailing list<br>
<a href="mailto:Swift-devel@ci.uchicago.edu" target="_blank">Swift-devel@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Swift-devel mailing list
<a href="mailto:Swift-devel@ci.uchicago.edu" target="_blank">Swift-devel@ci.uchicago.edu</a>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a>
</pre>
</blockquote>
<br>
</div></div><div class=""><pre cols="72">--
Michael Wilde
Mathematics and Computer Science Computation Institute
Argonne National Laboratory The University of Chicago
</pre>
</div></div>
<br>_______________________________________________<br>
Swift-devel mailing list<br>
<a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
<br></blockquote></div><br></div>