[Swift-devel] changes

[Mihael Hategan]

Hi,

I committed a fairly large number of changes. Some things got limited
testing, and some are work in progress. For example, I need to update
sites files used in tests, but it's getting late here.
Anyway, here's a summary:

1. New sites.xml format and removal of tc.data. I expect a bunch of
things to break, but we discussed cleaning both sites.xml and tc.data
many times and I guess I never got the courage to do it until now. This
format is not backwards compatible (intentionally). It goes like this:
<sites>
  <apps>? <!-- global apps -->
    <property name="...">...</property>* <!-- global app properties -->
    <env name="...">...</env>* <!-- global app envs -->
    <app name="..." executable="...">*
       <property/>*
       <env/>*
    </app>
  </apps>

  <site name="...">
    <execution provider="..."...>
      <property/>*
    </execution>
    <filesystem>?
     <property/>*
    </filesystem>
    <wokdirectory/>
    <scratch/>?
    <apps/>?
  <site>
</sites>

There are no more namespaces since they were mostly used to figure out
whether a property was going to the task or the site. I also changed
"profile" to "property", since "profile" seemed a bit arcane.

You should also now be able to use maxParallelTasks and
initialParallelTasks instead of jobThrottle and initialScore. It
computes the latter automatically.

2. Coasters now support multiple configurations. So you can have two
local:local coaster sites with different settings and it should work.
The way this works is that every configuration (and every different run)
gets its own job queue, settings, and block allocator, and they don't
interact with each other. This might be problematic with things like
slots which is meant to limit the number of jobs globally, but that's a
minor annoyance. 

3. Passive workers can be launched with a -c(oncurrency) argument (1 by
default). This will be the jobs per node setting. Each worker can have a
different number, and that should work as expected. For passive workers,
the client side jobsPerNode setting will be ignored.

4. There is a tool to analyze logs now (swift-log-info). This is simply
an offline version of the http monitor. It can parse logs and feed the
information to a web browser. It should also be able to follow logs as
they are produced and feed live information to a browser. It can also
fake a live log by parsing it slowly instead of all-at-once.

5. The http monitor got a bit of an update. You can look at all kinds of
statistics, browse through apps, look at pretty plots of how apps
behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
fashion. It should be cute and useful in one. Won't work with older
logs. Some pics attached.

6. There are now worker.pl "probes" for CPU usage, disk usage, and some
I/O stats. If you click on a link to a worker in the http monitor,
you'll get nice graphs of these things. The probes are currently
hardcoded to run every 60 seconds (although in the pictures below they
ran at 1/s).

Mihael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: s1.png
Type: image/png
Size: 65562 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/b97adb11/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: s2.png
Type: image/png
Size: 72654 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/b97adb11/attachment-0001.png>