[Swift-devel] changes

[Michael Wilde]

David,

Mihael suggested making the new config native to Swift, by converting 
the perl config code into Java.

That sounded pretty reasonable, and would make the format you devised be 
the sole external specification of all properties.

We were thinking that this could be done in 0.96. Thoughts?

- Mike

On 7/4/14, 6:05 PM, David Kelly wrote:
> This looks great - lots of very useful changes. I'll volunteer to make 
> the new config work with this sites.xml format.
>
>
> On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov 
> <mailto:hategan at mcs.anl.gov>> wrote:
>
>     Hi,
>
>     I committed a fairly large number of changes. Some things got limited
>     testing, and some are work in progress. For example, I need to update
>     sites files used in tests, but it's getting late here.
>     Anyway, here's a summary:
>
>     1. New sites.xml format and removal of tc.data. I expect a bunch of
>     things to break, but we discussed cleaning both sites.xml and tc.data
>     many times and I guess I never got the courage to do it until now.
>     This
>     format is not backwards compatible (intentionally). It goes like this:
>     <sites>
>       <apps>? <!-- global apps -->
>         <property name="...">...</property>* <!-- global app
>     properties -->
>         <env name="...">...</env>* <!-- global app envs -->
>         <app name="..." executable="...">*
>            <property/>*
>            <env/>*
>         </app>
>       </apps>
>
>       <site name="...">
>         <execution provider="..."...>
>           <property/>*
>         </execution>
>         <filesystem>?
>          <property/>*
>         </filesystem>
>         <wokdirectory/>
>         <scratch/>?
>         <apps/>?
>       <site>
>     </sites>
>
>     There are no more namespaces since they were mostly used to figure out
>     whether a property was going to the task or the site. I also changed
>     "profile" to "property", since "profile" seemed a bit arcane.
>
>     You should also now be able to use maxParallelTasks and
>     initialParallelTasks instead of jobThrottle and initialScore. It
>     computes the latter automatically.
>
>     2. Coasters now support multiple configurations. So you can have two
>     local:local coaster sites with different settings and it should work.
>     The way this works is that every configuration (and every
>     different run)
>     gets its own job queue, settings, and block allocator, and they don't
>     interact with each other. This might be problematic with things like
>     slots which is meant to limit the number of jobs globally, but
>     that's a
>     minor annoyance.
>
>     3. Passive workers can be launched with a -c(oncurrency) argument
>     (1 by
>     default). This will be the jobs per node setting. Each worker can
>     have a
>     different number, and that should work as expected. For passive
>     workers,
>     the client side jobsPerNode setting will be ignored.
>
>     4. There is a tool to analyze logs now (swift-log-info). This is
>     simply
>     an offline version of the http monitor. It can parse logs and feed the
>     information to a web browser. It should also be able to follow logs as
>     they are produced and feed live information to a browser. It can also
>     fake a live log by parsing it slowly instead of all-at-once.
>
>     5. The http monitor got a bit of an update. You can look at all
>     kinds of
>     statistics, browse through apps, look at pretty plots of how apps
>     behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
>     fashion. It should be cute and useful in one. Won't work with older
>     logs. Some pics attached.
>
>     6. There are now worker.pl <http://worker.pl> "probes" for CPU
>     usage, disk usage, and some
>     I/O stats. If you click on a link to a worker in the http monitor,
>     you'll get nice graphs of these things. The probes are currently
>     hardcoded to run every 60 seconds (although in the pictures below they
>     ran at 1/s).
>
>     Mihael
>
>     _______________________________________________
>     Swift-devel mailing list
>     Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
>     https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/19d7f236/attachment.html>