[Swift-devel] changes
Michael Wilde
wilde at anl.gov
Sat Jul 5 09:34:33 CDT 2014
This looks very good, deals with a lot of parsing issues, and is
licensed under Apache 2 (same as Swift).
But is it Java-only? If so, then down the road, for Swift/T with
Coaster C client, just parse options using a Java app perhaps?
- Mike
On 7/5/14, 12:58 AM, Mihael Hategan wrote:
> Thanks. I'll probably have questions as I go along.
>
> So when I spoke to Mike earlier I suggested YAML. After reading a bit I
> became convinced that YAML is not such a good idea. It's too complex and
> ambiguous at times.
>
> The unfortunate thing is that there is no good standardized hierarchical
> language for this sort of business except for XML. However, after some
> searching I found this: https://github.com/typesafehub/config
>
> It has a format that is a superset of JSON which allows a few nice
> syntactic changes that make it more user-friendly, such as not having to
> quote keys, the ability to use ":" and "=" interchangeably, property/env
> var expansion, and the ability to omit the separating comma if there is
> a new line. For example:
>
> foo {
> bar = 10
> baz = 12
> }
>
> This is I believe very close to David's format.
>
> But there is something else, which I like, and that is include
> directives. I think we ran into a number of problems with the fact that
> Swift loads various configurations and that can be surprising to a user.
> At the same time, some users do want to have some default settings and
> only override a few.
>
> I believe that includes address this problem elegantly in that users can
> use common configurations and override a few things, but it will always
> be obvious when looking at a config file that there are things coming
> from another file.
>
> Mihael
>
> On Fri, 2014-07-04 at 18:45 -0500, David Kelly wrote:
>> Great, I think that's a good idea. Most of how it works should be described
>> in the trunk userguide - how to handle run directories, config syntax,
>> search paths for swift.properties, templating, etc.
>>
>>
>> On Fri, Jul 4, 2014 at 6:21 PM, Michael Wilde <wilde at anl.gov> wrote:
>>
>>> David,
>>>
>>> Mihael suggested making the new config native to Swift, by converting the
>>> perl config code into Java.
>>>
>>> That sounded pretty reasonable, and would make the format you devised be
>>> the sole external specification of all properties.
>>>
>>> We were thinking that this could be done in 0.96. Thoughts?
>>>
>>> - Mike
>>>
>>>
>>> On 7/4/14, 6:05 PM, David Kelly wrote:
>>>
>>> This looks great - lots of very useful changes. I'll volunteer to make the
>>> new config work with this sites.xml format.
>>>
>>>
>>> On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I committed a fairly large number of changes. Some things got limited
>>>> testing, and some are work in progress. For example, I need to update
>>>> sites files used in tests, but it's getting late here.
>>>> Anyway, here's a summary:
>>>>
>>>> 1. New sites.xml format and removal of tc.data. I expect a bunch of
>>>> things to break, but we discussed cleaning both sites.xml and tc.data
>>>> many times and I guess I never got the courage to do it until now. This
>>>> format is not backwards compatible (intentionally). It goes like this:
>>>> <sites>
>>>> <apps>? <!-- global apps -->
>>>> <property name="...">...</property>* <!-- global app properties -->
>>>> <env name="...">...</env>* <!-- global app envs -->
>>>> <app name="..." executable="...">*
>>>> <property/>*
>>>> <env/>*
>>>> </app>
>>>> </apps>
>>>>
>>>> <site name="...">
>>>> <execution provider="..."...>
>>>> <property/>*
>>>> </execution>
>>>> <filesystem>?
>>>> <property/>*
>>>> </filesystem>
>>>> <wokdirectory/>
>>>> <scratch/>?
>>>> <apps/>?
>>>> <site>
>>>> </sites>
>>>>
>>>> There are no more namespaces since they were mostly used to figure out
>>>> whether a property was going to the task or the site. I also changed
>>>> "profile" to "property", since "profile" seemed a bit arcane.
>>>>
>>>> You should also now be able to use maxParallelTasks and
>>>> initialParallelTasks instead of jobThrottle and initialScore. It
>>>> computes the latter automatically.
>>>>
>>>> 2. Coasters now support multiple configurations. So you can have two
>>>> local:local coaster sites with different settings and it should work.
>>>> The way this works is that every configuration (and every different run)
>>>> gets its own job queue, settings, and block allocator, and they don't
>>>> interact with each other. This might be problematic with things like
>>>> slots which is meant to limit the number of jobs globally, but that's a
>>>> minor annoyance.
>>>>
>>>> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
>>>> default). This will be the jobs per node setting. Each worker can have a
>>>> different number, and that should work as expected. For passive workers,
>>>> the client side jobsPerNode setting will be ignored.
>>>>
>>>> 4. There is a tool to analyze logs now (swift-log-info). This is simply
>>>> an offline version of the http monitor. It can parse logs and feed the
>>>> information to a web browser. It should also be able to follow logs as
>>>> they are produced and feed live information to a browser. It can also
>>>> fake a live log by parsing it slowly instead of all-at-once.
>>>>
>>>> 5. The http monitor got a bit of an update. You can look at all kinds of
>>>> statistics, browse through apps, look at pretty plots of how apps
>>>> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
>>>> fashion. It should be cute and useful in one. Won't work with older
>>>> logs. Some pics attached.
>>>>
>>>> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
>>>> I/O stats. If you click on a link to a worker in the http monitor,
>>>> you'll get nice graphs of these things. The probes are currently
>>>> hardcoded to run every 60 seconds (although in the pictures below they
>>>> ran at 1/s).
>>>>
>>>> Mihael
>>>>
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>>
>>> --
>>> Michael Wilde
>>> Mathematics and Computer Science Computation Institute
>>> Argonne National Laboratory The University of Chicago
>>>
>>>
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
--
Michael Wilde
Mathematics and Computer Science Computation Institute
Argonne National Laboratory The University of Chicago
More information about the Swift-devel
mailing list