<div dir="ltr">This looks great - lots of very useful changes. I'll volunteer to make the new config work with this sites.xml format.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <span dir="ltr"><<a href="mailto:hategan@mcs.anl.gov" target="_blank">hategan@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I committed a fairly large number of changes. Some things got limited<br>
testing, and some are work in progress. For example, I need to update<br>
sites files used in tests, but it's getting late here.<br>
Anyway, here's a summary:<br>
<br>
1. New sites.xml format and removal of tc.data. I expect a bunch of<br>
things to break, but we discussed cleaning both sites.xml and tc.data<br>
many times and I guess I never got the courage to do it until now. This<br>
format is not backwards compatible (intentionally). It goes like this:<br>
<sites><br>
<apps>? <!-- global apps --><br>
<property name="...">...</property>* <!-- global app properties --><br>
<env name="...">...</env>* <!-- global app envs --><br>
<app name="..." executable="...">*<br>
<property/>*<br>
<env/>*<br>
</app><br>
</apps><br>
<br>
<site name="..."><br>
<execution provider="..."...><br>
<property/>*<br>
</execution><br>
<filesystem>?<br>
<property/>*<br>
</filesystem><br>
<wokdirectory/><br>
<scratch/>?<br>
<apps/>?<br>
<site><br>
</sites><br>
<br>
There are no more namespaces since they were mostly used to figure out<br>
whether a property was going to the task or the site. I also changed<br>
"profile" to "property", since "profile" seemed a bit arcane.<br>
<br>
You should also now be able to use maxParallelTasks and<br>
initialParallelTasks instead of jobThrottle and initialScore. It<br>
computes the latter automatically.<br>
<br>
2. Coasters now support multiple configurations. So you can have two<br>
local:local coaster sites with different settings and it should work.<br>
The way this works is that every configuration (and every different run)<br>
gets its own job queue, settings, and block allocator, and they don't<br>
interact with each other. This might be problematic with things like<br>
slots which is meant to limit the number of jobs globally, but that's a<br>
minor annoyance.<br>
<br>
3. Passive workers can be launched with a -c(oncurrency) argument (1 by<br>
default). This will be the jobs per node setting. Each worker can have a<br>
different number, and that should work as expected. For passive workers,<br>
the client side jobsPerNode setting will be ignored.<br>
<br>
4. There is a tool to analyze logs now (swift-log-info). This is simply<br>
an offline version of the http monitor. It can parse logs and feed the<br>
information to a web browser. It should also be able to follow logs as<br>
they are produced and feed live information to a browser. It can also<br>
fake a live log by parsing it slowly instead of all-at-once.<br>
<br>
5. The http monitor got a bit of an update. You can look at all kinds of<br>
statistics, browse through apps, look at pretty plots of how apps<br>
behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked<br>
fashion. It should be cute and useful in one. Won't work with older<br>
logs. Some pics attached.<br>
<br>
6. There are now <a href="http://worker.pl" target="_blank">worker.pl</a> "probes" for CPU usage, disk usage, and some<br>
I/O stats. If you click on a link to a worker in the http monitor,<br>
you'll get nice graphs of these things. The probes are currently<br>
hardcoded to run every 60 seconds (although in the pictures below they<br>
ran at 1/s).<br>
<span class="HOEnZb"><font color="#888888"><br>
Mihael<br>
</font></span><br>_______________________________________________<br>
Swift-devel mailing list<br>
<a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
<br></blockquote></div><br></div>