[Swift-devel] changes

[David Kelly]

This looks great - lots of very useful changes. I'll volunteer to make the
new config work with this sites.xml format.


On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Hi,
>
> I committed a fairly large number of changes. Some things got limited
> testing, and some are work in progress. For example, I need to update
> sites files used in tests, but it's getting late here.
> Anyway, here's a summary:
>
> 1. New sites.xml format and removal of tc.data. I expect a bunch of
> things to break, but we discussed cleaning both sites.xml and tc.data
> many times and I guess I never got the courage to do it until now. This
> format is not backwards compatible (intentionally). It goes like this:
> <sites>
>   <apps>? <!-- global apps -->
>     <property name="...">...</property>* <!-- global app properties -->
>     <env name="...">...</env>* <!-- global app envs -->
>     <app name="..." executable="...">*
>        <property/>*
>        <env/>*
>     </app>
>   </apps>
>
>   <site name="...">
>     <execution provider="..."...>
>       <property/>*
>     </execution>
>     <filesystem>?
>      <property/>*
>     </filesystem>
>     <wokdirectory/>
>     <scratch/>?
>     <apps/>?
>   <site>
> </sites>
>
> There are no more namespaces since they were mostly used to figure out
> whether a property was going to the task or the site. I also changed
> "profile" to "property", since "profile" seemed a bit arcane.
>
> You should also now be able to use maxParallelTasks and
> initialParallelTasks instead of jobThrottle and initialScore. It
> computes the latter automatically.
>
> 2. Coasters now support multiple configurations. So you can have two
> local:local coaster sites with different settings and it should work.
> The way this works is that every configuration (and every different run)
> gets its own job queue, settings, and block allocator, and they don't
> interact with each other. This might be problematic with things like
> slots which is meant to limit the number of jobs globally, but that's a
> minor annoyance.
>
> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> default). This will be the jobs per node setting. Each worker can have a
> different number, and that should work as expected. For passive workers,
> the client side jobsPerNode setting will be ignored.
>
> 4. There is a tool to analyze logs now (swift-log-info). This is simply
> an offline version of the http monitor. It can parse logs and feed the
> information to a web browser. It should also be able to follow logs as
> they are produced and feed live information to a browser. It can also
> fake a live log by parsing it slowly instead of all-at-once.
>
> 5. The http monitor got a bit of an update. You can look at all kinds of
> statistics, browse through apps, look at pretty plots of how apps
> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> fashion. It should be cute and useful in one. Won't work with older
> logs. Some pics attached.
>
> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> I/O stats. If you click on a link to a worker in the http monitor,
> you'll get nice graphs of these things. The probes are currently
> hardcoded to run every 60 seconds (although in the pictures below they
> ran at 1/s).
>
> Mihael
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/55fb86b0/attachment.html>