[Swift-devel] changes

[David Kelly]

Great, I think that's a good idea. Most of how it works should be described
in the trunk userguide - how to handle run directories, config syntax,
search paths for swift.properties, templating, etc.


On Fri, Jul 4, 2014 at 6:21 PM, Michael Wilde <wilde at anl.gov> wrote:

>  David,
>
> Mihael suggested making the new config native to Swift, by converting the
> perl config code into Java.
>
> That sounded pretty reasonable, and would make the format you devised be
> the sole external specification of all properties.
>
> We were thinking that this could be done in 0.96. Thoughts?
>
> - Mike
>
>
> On 7/4/14, 6:05 PM, David Kelly wrote:
>
> This looks great - lots of very useful changes. I'll volunteer to make the
> new config work with this sites.xml format.
>
>
> On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>
>> Hi,
>>
>> I committed a fairly large number of changes. Some things got limited
>> testing, and some are work in progress. For example, I need to update
>> sites files used in tests, but it's getting late here.
>> Anyway, here's a summary:
>>
>> 1. New sites.xml format and removal of tc.data. I expect a bunch of
>> things to break, but we discussed cleaning both sites.xml and tc.data
>> many times and I guess I never got the courage to do it until now. This
>> format is not backwards compatible (intentionally). It goes like this:
>> <sites>
>>   <apps>? <!-- global apps -->
>>     <property name="...">...</property>* <!-- global app properties -->
>>     <env name="...">...</env>* <!-- global app envs -->
>>     <app name="..." executable="...">*
>>        <property/>*
>>        <env/>*
>>     </app>
>>   </apps>
>>
>>   <site name="...">
>>     <execution provider="..."...>
>>       <property/>*
>>     </execution>
>>     <filesystem>?
>>      <property/>*
>>     </filesystem>
>>     <wokdirectory/>
>>     <scratch/>?
>>     <apps/>?
>>   <site>
>> </sites>
>>
>> There are no more namespaces since they were mostly used to figure out
>> whether a property was going to the task or the site. I also changed
>> "profile" to "property", since "profile" seemed a bit arcane.
>>
>> You should also now be able to use maxParallelTasks and
>> initialParallelTasks instead of jobThrottle and initialScore. It
>> computes the latter automatically.
>>
>> 2. Coasters now support multiple configurations. So you can have two
>> local:local coaster sites with different settings and it should work.
>> The way this works is that every configuration (and every different run)
>> gets its own job queue, settings, and block allocator, and they don't
>> interact with each other. This might be problematic with things like
>> slots which is meant to limit the number of jobs globally, but that's a
>> minor annoyance.
>>
>> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
>> default). This will be the jobs per node setting. Each worker can have a
>> different number, and that should work as expected. For passive workers,
>> the client side jobsPerNode setting will be ignored.
>>
>> 4. There is a tool to analyze logs now (swift-log-info). This is simply
>> an offline version of the http monitor. It can parse logs and feed the
>> information to a web browser. It should also be able to follow logs as
>> they are produced and feed live information to a browser. It can also
>> fake a live log by parsing it slowly instead of all-at-once.
>>
>> 5. The http monitor got a bit of an update. You can look at all kinds of
>> statistics, browse through apps, look at pretty plots of how apps
>> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
>> fashion. It should be cute and useful in one. Won't work with older
>> logs. Some pics attached.
>>
>> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
>> I/O stats. If you click on a link to a worker in the http monitor,
>> you'll get nice graphs of these things. The probes are currently
>> hardcoded to run every 60 seconds (although in the pictures below they
>> ran at 1/s).
>>
>> Mihael
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>
>>
>
>
> _______________________________________________
> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
> --
> Michael Wilde
> Mathematics and Computer Science          Computation Institute
> Argonne National Laboratory               The University of Chicago
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/2f067e2f/attachment.html>