[Swift-devel] Swiftconfig merge / changes

[Mihael Hategan]

On Wed, 2011-02-02 at 13:22 -0800, Mihael Hategan wrote:
> On Wed, 2011-02-02 at 00:04 -0600, jon.monette at gmail.com wrote:
> > I agree switching the swiftrun/swiftconfig to using bash may not be
> > the best idea due to the above reasons David has mentioned. However I
> > do not believe switching over to using Java is the solution. In the
> > end swiftrun is calling swift which is a shell script. Having a java
> > process call a shell script which in turns starts java processes may
> > not be the best plan of action.
> 
> That doesn't need to be so. A java program can invoke the main() method
> of swift.

In fact, this may very well turn out to fit into that swift shell model
that Mike mentioned that would keep coaster workers (and other JVM
persistent things) alive between swift runs.

> 
> >  May I suggest using python? Not much changes between python versions
> > and it has very good xml libraries to work with. I do not believe
> > there will be very many comparability issues in using python but in
> > all programming there are always these issues. Python is also much
> > easier to read in my humble opinion.
> > 
> > ----- Reply message -----
> > From: "David Kelly" <dk0966 at cs.ship.edu>
> > Date: Tue, Feb 1, 2011 11:24 pm
> > Subject: [Swift-devel] Swiftconfig merge / changes
> > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > Cc: "swift-devel" <swift-devel at ci.uchicago.edu>
> > 
> > 
> > 
> > 
> > On Tue, Feb 1, 2011 at 12:07 PM, Michael Wilde <wilde at mcs.anl.gov>
> > wrote:
> > 
> >         Also, Im very eager to reconcile and unify into one
> >         coordinated plan the idea of merging the best aspects of
> >         Justin's ad-hoc configuration methid, which he's documented on
> >         the SWFT wiki (see eg CoasterCookbook etc) and your
> >         swiftconfig/swiftrun mechanism.
> >         
> >         I like the fact that Justin's is based on sh rather then perl
> >         - that will make maintenance easier and reduce portability
> >         issues. But I also want to bring back various concepts from
> >         swiftconfig/run into this mechanism.  If we can agree on a
> >         spec, perhaps that would be a good project for you to (resume)
> >         work on?
> > 
> > Here is what I see as the list of features we would need to add to the
> > shell script in order to merge features:
> > 
> > From swiftconfig:
> > 
> > - XML editor. It should have some basic knowledge of valid inputs (for
> > example, when you're editing the execution provider setting, it will
> > only allow you to select one that swift knows)
> > - Editor for adding/removing/modifying apps to the tc file and
> > verifying formatting
> > - Using templates to generate new site configurations
> > - Using templates to generate new templates
> > - Importing existing configuration files into the template
> > format/directory structure
> > - Creating and managing ssh configurations for various hosts
> > 
> > From swiftrun:
> > 
> > - A new swiftrun mechanism which reads from a template directory,
> > generates the run.XXXX directory, links input data, and runs swift
> > - The concept of site groups. For example, creating a group called
> > "MCS-coasters" which combines the individual configurations of
> > thrash-coasters, thwomp-coasters, etc.
> > - Similar to above, application sets for handling different sets of
> > apps
> > 
> > It's possible we may not need or want all of these. I just wanted to
> > point out what would be involved if we merged the main features as
> > they currently exist in the swiftconfig/swiftrun utilities. I also
> > have a list of suggestions for future improvements which include
> > things like revision control for modifications and a tagging/search
> > system for searching through previous swift runs.
> > 
> > The suggestion was moving away from perl due to portability issues. I
> > can understand that, but what do you think about doing it in Java?
> > Bash is nice, but in some ways it suffers from the same portability
> > issues that perl does. Here are a few issues I have run into so far.
> > The "swift" shell script we use calls /bin/sh. Sometimes this is bash.
> > On some linux systems it's actually a shell called ash, which is
> > similar but does not always compatible. If you can assume there will
> > always be a /bin/bash, there are individual differences between bash
> > versions. When I was testing changes to usage stats, I ran into a
> > really old version of bash which caused it to fail. Then there are
> > things like /dev/udp which may or may not be turned on 
> > based on bash compilation options. Bash is nice, but it's a little
> > limited in what is can do by itself without relying on a lot of
> > external applications.. which may not be always be installed on the
> > system or work as expected.
> > 
> > The nice thing about doing it in Java is that you can bank on Java 1.5
> > or later being there regardless of whatever system quirks you may run
> > into. Java also includes XML handling libraries and has nice regular
> > expression system which would make this a little easier to handle. It
> > will also be easier to manage as complexity increases in the future.
> > 
> > Shifting gears a bit.. what I would _really_ would like in the future
> > is a graphical interface on top of swiftconfig and swiftrun. I may be
> > the only one, but I haven't given up on this idea :-) 
> > 
> > - Have one visual interface running locally which will let me do
> > everything
> > - Easily and visually configure swift to submit jobs on a remote
> > system
> > - Visually create a workflow which generates a swift script
> > - Select a swift script and be able to easily change parameters,
> > queue, maxtime, # nodes, etc on the fly
> > - See a visual progression of what is happening. Something like a
> > swing version of -tui
> > 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel