[Swift-devel] changes

[Mihael Hategan]

No, but I think this is a good opportunity for me to write one.

Mihael

On Fri, 2014-07-04 at 11:49 -0500, Tim Armstrong wrote:
> I had a semi-related question: is there any documentation aside from the
> user guide about sites, providers, and job managers in Swift/Coasters?
> 
> - Tim
> 
> 
> On Fri, Jul 4, 2014 at 11:02 AM, Michael Wilde <wilde at anl.gov> wrote:
> 
> >  A lot of great work, here, Mihael - very nice!
> >
> > We'll need to re-adjust the new config mechanism to match this, unless
> > you've already done that.
> >
> > This means that we'll need to get all users converted to the new config
> > before this gets released - else they will have no backwards compatibility
> > for their current configurations (sites and tc files).
> >
> > Nonetheless, as we discussed in a prior meeting, I think we agreed that
> > this was a necessary step and the right way to go.
> >
> > - Mike
> >
> >
> > On 7/4/14, 3:07 AM, Mihael Hategan wrote:
> >
> > Hi,
> >
> > I committed a fairly large number of changes. Some things got limited
> > testing, and some are work in progress. For example, I need to update
> > sites files used in tests, but it's getting late here.
> > Anyway, here's a summary:
> >
> > 1. New sites.xml format and removal of tc.data. I expect a bunch of
> > things to break, but we discussed cleaning both sites.xml and tc.data
> > many times and I guess I never got the courage to do it until now. This
> > format is not backwards compatible (intentionally). It goes like this:
> > <sites>
> >   <apps>? <!-- global apps -->
> >     <property name="...">...</property>* <!-- global app properties -->
> >     <env name="...">...</env>* <!-- global app envs -->
> >     <app name="..." executable="...">*
> >        <property/>*
> >        <env/>*
> >     </app>
> >   </apps>
> >
> >   <site name="...">
> >     <execution provider="..."...>
> >       <property/>*
> >     </execution>
> >     <filesystem>?
> >      <property/>*
> >     </filesystem>
> >     <wokdirectory/>
> >     <scratch/>?
> >     <apps/>?
> >   <site>
> > </sites>
> >
> > There are no more namespaces since they were mostly used to figure out
> > whether a property was going to the task or the site. I also changed
> > "profile" to "property", since "profile" seemed a bit arcane.
> >
> > You should also now be able to use maxParallelTasks and
> > initialParallelTasks instead of jobThrottle and initialScore. It
> > computes the latter automatically.
> >
> > 2. Coasters now support multiple configurations. So you can have two
> > local:local coaster sites with different settings and it should work.
> > The way this works is that every configuration (and every different run)
> > gets its own job queue, settings, and block allocator, and they don't
> > interact with each other. This might be problematic with things like
> > slots which is meant to limit the number of jobs globally, but that's a
> > minor annoyance.
> >
> > 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> > default). This will be the jobs per node setting. Each worker can have a
> > different number, and that should work as expected. For passive workers,
> > the client side jobsPerNode setting will be ignored.
> >
> > 4. There is a tool to analyze logs now (swift-log-info). This is simply
> > an offline version of the http monitor. It can parse logs and feed the
> > information to a web browser. It should also be able to follow logs as
> > they are produced and feed live information to a browser. It can also
> > fake a live log by parsing it slowly instead of all-at-once.
> >
> > 5. The http monitor got a bit of an update. You can look at all kinds of
> > statistics, browse through apps, look at pretty plots of how apps
> > behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> > fashion. It should be cute and useful in one. Won't work with older
> > logs. Some pics attached.
> >
> > 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> > I/O stats. If you click on a link to a worker in the http monitor,
> > you'll get nice graphs of these things. The probes are currently
> > hardcoded to run every 60 seconds (although in the pictures below they
> > ran at 1/s).
> >
> > Mihael
> >
> >
> >
> > _______________________________________________
> > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> > --
> > Michael Wilde
> > Mathematics and Computer Science          Computation Institute
> > Argonne National Laboratory               The University of Chicago
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel