[Swift-devel] Swiftconfig merge / changes

[Mihael Hategan]

On Wed, 2011-02-02 at 00:04 -0600, jon.monette at gmail.com wrote:
> I agree switching the swiftrun/swiftconfig to using bash may not be
> the best idea due to the above reasons David has mentioned. However I
> do not believe switching over to using Java is the solution. In the
> end swiftrun is calling swift which is a shell script. Having a java
> process call a shell script which in turns starts java processes may
> not be the best plan of action.

That doesn't need to be so. A java program can invoke the main() method
of swift.

>  May I suggest using python? Not much changes between python versions
> and it has very good xml libraries to work with. I do not believe
> there will be very many comparability issues in using python but in
> all programming there are always these issues. Python is also much
> easier to read in my humble opinion.
> 
> ----- Reply message -----
> From: "David Kelly" <dk0966 at cs.ship.edu>
> Date: Tue, Feb 1, 2011 11:24 pm
> Subject: [Swift-devel] Swiftconfig merge / changes
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "swift-devel" <swift-devel at ci.uchicago.edu>
> 
> 
> 
> 
> On Tue, Feb 1, 2011 at 12:07 PM, Michael Wilde <wilde at mcs.anl.gov>
> wrote:
> 
>         Also, Im very eager to reconcile and unify into one
>         coordinated plan the idea of merging the best aspects of
>         Justin's ad-hoc configuration methid, which he's documented on
>         the SWFT wiki (see eg CoasterCookbook etc) and your
>         swiftconfig/swiftrun mechanism.
>         
>         I like the fact that Justin's is based on sh rather then perl
>         - that will make maintenance easier and reduce portability
>         issues. But I also want to bring back various concepts from
>         swiftconfig/run into this mechanism.  If we can agree on a
>         spec, perhaps that would be a good project for you to (resume)
>         work on?
> 
> Here is what I see as the list of features we would need to add to the
> shell script in order to merge features:
> 
> From swiftconfig:
> 
> - XML editor. It should have some basic knowledge of valid inputs (for
> example, when you're editing the execution provider setting, it will
> only allow you to select one that swift knows)
> - Editor for adding/removing/modifying apps to the tc file and
> verifying formatting
> - Using templates to generate new site configurations
> - Using templates to generate new templates
> - Importing existing configuration files into the template
> format/directory structure
> - Creating and managing ssh configurations for various hosts
> 
> From swiftrun:
> 
> - A new swiftrun mechanism which reads from a template directory,
> generates the run.XXXX directory, links input data, and runs swift
> - The concept of site groups. For example, creating a group called
> "MCS-coasters" which combines the individual configurations of
> thrash-coasters, thwomp-coasters, etc.
> - Similar to above, application sets for handling different sets of
> apps
> 
> It's possible we may not need or want all of these. I just wanted to
> point out what would be involved if we merged the main features as
> they currently exist in the swiftconfig/swiftrun utilities. I also
> have a list of suggestions for future improvements which include
> things like revision control for modifications and a tagging/search
> system for searching through previous swift runs.
> 
> The suggestion was moving away from perl due to portability issues. I
> can understand that, but what do you think about doing it in Java?
> Bash is nice, but in some ways it suffers from the same portability
> issues that perl does. Here are a few issues I have run into so far.
> The "swift" shell script we use calls /bin/sh. Sometimes this is bash.
> On some linux systems it's actually a shell called ash, which is
> similar but does not always compatible. If you can assume there will
> always be a /bin/bash, there are individual differences between bash
> versions. When I was testing changes to usage stats, I ran into a
> really old version of bash which caused it to fail. Then there are
> things like /dev/udp which may or may not be turned on 
> based on bash compilation options. Bash is nice, but it's a little
> limited in what is can do by itself without relying on a lot of
> external applications.. which may not be always be installed on the
> system or work as expected.
> 
> The nice thing about doing it in Java is that you can bank on Java 1.5
> or later being there regardless of whatever system quirks you may run
> into. Java also includes XML handling libraries and has nice regular
> expression system which would make this a little easier to handle. It
> will also be easier to manage as complexity increases in the future.
> 
> Shifting gears a bit.. what I would _really_ would like in the future
> is a graphical interface on top of swiftconfig and swiftrun. I may be
> the only one, but I haven't given up on this idea :-) 
> 
> - Have one visual interface running locally which will let me do
> everything
> - Easily and visually configure swift to submit jobs on a remote
> system
> - Visually create a workflow which generates a swift script
> - Select a swift script and be able to easily change parameters,
> queue, maxtime, # nodes, etc on the fly
> - See a visual progression of what is happening. Something like a
> swing version of -tui
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel