[Swift-devel] changes

[Michael Wilde]

A lot of great work, here, Mihael - very nice!

We'll need to re-adjust the new config mechanism to match this, unless 
you've already done that.

This means that we'll need to get all users converted to the new config 
before this gets released - else they will have no backwards 
compatibility for their current configurations (sites and tc files).

Nonetheless, as we discussed in a prior meeting, I think we agreed that 
this was a necessary step and the right way to go.

- Mike

On 7/4/14, 3:07 AM, Mihael Hategan wrote:
> Hi,
>
> I committed a fairly large number of changes. Some things got limited
> testing, and some are work in progress. For example, I need to update
> sites files used in tests, but it's getting late here.
> Anyway, here's a summary:
>
> 1. New sites.xml format and removal of tc.data. I expect a bunch of
> things to break, but we discussed cleaning both sites.xml and tc.data
> many times and I guess I never got the courage to do it until now. This
> format is not backwards compatible (intentionally). It goes like this:
> <sites>
>    <apps>? <!-- global apps -->
>      <property name="...">...</property>* <!-- global app properties -->
>      <env name="...">...</env>* <!-- global app envs -->
>      <app name="..." executable="...">*
>         <property/>*
>         <env/>*
>      </app>
>    </apps>
>
>    <site name="...">
>      <execution provider="..."...>
>        <property/>*
>      </execution>
>      <filesystem>?
>       <property/>*
>      </filesystem>
>      <wokdirectory/>
>      <scratch/>?
>      <apps/>?
>    <site>
> </sites>
>
> There are no more namespaces since they were mostly used to figure out
> whether a property was going to the task or the site. I also changed
> "profile" to "property", since "profile" seemed a bit arcane.
>
> You should also now be able to use maxParallelTasks and
> initialParallelTasks instead of jobThrottle and initialScore. It
> computes the latter automatically.
>
> 2. Coasters now support multiple configurations. So you can have two
> local:local coaster sites with different settings and it should work.
> The way this works is that every configuration (and every different run)
> gets its own job queue, settings, and block allocator, and they don't
> interact with each other. This might be problematic with things like
> slots which is meant to limit the number of jobs globally, but that's a
> minor annoyance.
>
> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> default). This will be the jobs per node setting. Each worker can have a
> different number, and that should work as expected. For passive workers,
> the client side jobsPerNode setting will be ignored.
>
> 4. There is a tool to analyze logs now (swift-log-info). This is simply
> an offline version of the http monitor. It can parse logs and feed the
> information to a web browser. It should also be able to follow logs as
> they are produced and feed live information to a browser. It can also
> fake a live log by parsing it slowly instead of all-at-once.
>
> 5. The http monitor got a bit of an update. You can look at all kinds of
> statistics, browse through apps, look at pretty plots of how apps
> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> fashion. It should be cute and useful in one. Won't work with older
> logs. Some pics attached.
>
> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> I/O stats. If you click on a link to a worker in the http monitor,
> you'll get nice graphs of these things. The probes are currently
> hardcoded to run every 60 seconds (although in the pictures below they
> ran at 1/s).
>
> Mihael
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/d9007f9f/attachment.html>