[Swift-devel] changes

Mihael Hategan hategan at mcs.anl.gov
Sat Jul 5 00:58:02 CDT 2014 

Thanks. I'll probably have questions as I go along.

So when I spoke to Mike earlier I suggested YAML. After reading a bit I
became convinced that YAML is not such a good idea. It's too complex and
ambiguous at times.

The unfortunate thing is that there is no good standardized hierarchical
language for this sort of business except for XML. However, after some
searching I found this: https://github.com/typesafehub/config

It has a format that is a superset of JSON which allows a few nice
syntactic changes that make it more user-friendly, such as not having to
quote keys, the ability to use ":" and "=" interchangeably, property/env
var expansion, and the ability to omit the separating comma if there is
a new line. For example:

foo {
    bar = 10
    baz = 12
}

This is I believe very close to David's format.

But there is something else, which I like, and that is include
directives. I think we ran into a number of problems with the fact that
Swift loads various configurations and that can be surprising to a user.
At the same time, some users do want to have some default settings and
only override a few.

I believe that includes address this problem elegantly in that users can
use common configurations and override a few things, but it will always
be obvious when looking at a config file that there are things coming
from another file.

Mihael

On Fri, 2014-07-04 at 18:45 -0500, David Kelly wrote:
> Great, I think that's a good idea. Most of how it works should be described
> in the trunk userguide - how to handle run directories, config syntax,
> search paths for swift.properties, templating, etc.
> 
> 
> On Fri, Jul 4, 2014 at 6:21 PM, Michael Wilde <wilde at anl.gov> wrote:
> 
> >  David,
> >
> > Mihael suggested making the new config native to Swift, by converting the
> > perl config code into Java.
> >
> > That sounded pretty reasonable, and would make the format you devised be
> > the sole external specification of all properties.
> >
> > We were thinking that this could be done in 0.96. Thoughts?
> >
> > - Mike
> >
> >
> > On 7/4/14, 6:05 PM, David Kelly wrote:
> >
> > This looks great - lots of very useful changes. I'll volunteer to make the
> > new config work with this sites.xml format.
> >
> >
> > On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >
> >> Hi,
> >>
> >> I committed a fairly large number of changes. Some things got limited
> >> testing, and some are work in progress. For example, I need to update
> >> sites files used in tests, but it's getting late here.
> >> Anyway, here's a summary:
> >>
> >> 1. New sites.xml format and removal of tc.data. I expect a bunch of
> >> things to break, but we discussed cleaning both sites.xml and tc.data
> >> many times and I guess I never got the courage to do it until now. This
> >> format is not backwards compatible (intentionally). It goes like this:
> >> <sites>
> >>   <apps>? <!-- global apps -->
> >>     <property name="...">...</property>* <!-- global app properties -->
> >>     <env name="...">...</env>* <!-- global app envs -->
> >>     <app name="..." executable="...">*
> >>        <property/>*
> >>        <env/>*
> >>     </app>
> >>   </apps>
> >>
> >>   <site name="...">
> >>     <execution provider="..."...>
> >>       <property/>*
> >>     </execution>
> >>     <filesystem>?
> >>      <property/>*
> >>     </filesystem>
> >>     <wokdirectory/>
> >>     <scratch/>?
> >>     <apps/>?
> >>   <site>
> >> </sites>
> >>
> >> There are no more namespaces since they were mostly used to figure out
> >> whether a property was going to the task or the site. I also changed
> >> "profile" to "property", since "profile" seemed a bit arcane.
> >>
> >> You should also now be able to use maxParallelTasks and
> >> initialParallelTasks instead of jobThrottle and initialScore. It
> >> computes the latter automatically.
> >>
> >> 2. Coasters now support multiple configurations. So you can have two
> >> local:local coaster sites with different settings and it should work.
> >> The way this works is that every configuration (and every different run)
> >> gets its own job queue, settings, and block allocator, and they don't
> >> interact with each other. This might be problematic with things like
> >> slots which is meant to limit the number of jobs globally, but that's a
> >> minor annoyance.
> >>
> >> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> >> default). This will be the jobs per node setting. Each worker can have a
> >> different number, and that should work as expected. For passive workers,
> >> the client side jobsPerNode setting will be ignored.
> >>
> >> 4. There is a tool to analyze logs now (swift-log-info). This is simply
> >> an offline version of the http monitor. It can parse logs and feed the
> >> information to a web browser. It should also be able to follow logs as
> >> they are produced and feed live information to a browser. It can also
> >> fake a live log by parsing it slowly instead of all-at-once.
> >>
> >> 5. The http monitor got a bit of an update. You can look at all kinds of
> >> statistics, browse through apps, look at pretty plots of how apps
> >> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> >> fashion. It should be cute and useful in one. Won't work with older
> >> logs. Some pics attached.
> >>
> >> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> >> I/O stats. If you click on a link to a worker in the http monitor,
> >> you'll get nice graphs of these things. The probes are currently
> >> hardcoded to run every 60 seconds (although in the pictures below they
> >> ran at 1/s).
> >>
> >> Mihael
> >>
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>
> >>
> >
> >
> > _______________________________________________
> > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> > --
> > Michael Wilde
> > Mathematics and Computer Science          Computation Institute
> > Argonne National Laboratory               The University of Chicago
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

More information about the Swift-devel mailing list