[Swift-devel] Swiftconfig merge / changes

[David Kelly]

On Tue, Feb 1, 2011 at 12:07 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

Also, Im very eager to reconcile and unify into one coordinated plan the
> idea of merging the best aspects of Justin's ad-hoc configuration methid,
> which he's documented on the SWFT wiki (see eg CoasterCookbook etc) and your
> swiftconfig/swiftrun mechanism.
>
> I like the fact that Justin's is based on sh rather then perl - that will
> make maintenance easier and reduce portability issues. But I also want to
> bring back various concepts from swiftconfig/run into this mechanism.  If we
> can agree on a spec, perhaps that would be a good project for you to
> (resume) work on?
>

Here is what I see as the list of features we would need to add to the shell
script in order to merge features:

>From swiftconfig:

- XML editor. It should have some basic knowledge of valid inputs (for
example, when you're editing the execution provider setting, it will only
allow you to select one that swift knows)
- Editor for adding/removing/modifying apps to the tc file and verifying
formatting
- Using templates to generate new site configurations
- Using templates to generate new templates
- Importing existing configuration files into the template format/directory
structure
- Creating and managing ssh configurations for various hosts

>From swiftrun:

- A new swiftrun mechanism which reads from a template directory, generates
the run.XXXX directory, links input data, and runs swift
- The concept of site groups. For example, creating a group called
"MCS-coasters" which combines the individual configurations of
thrash-coasters, thwomp-coasters, etc.
- Similar to above, application sets for handling different sets of apps

It's possible we may not need or want all of these. I just wanted to point
out what would be involved if we merged the main features as they currently
exist in the swiftconfig/swiftrun utilities. I also have a list of
suggestions for future improvements which include things like revision
control for modifications and a tagging/search system for searching through
previous swift runs.

The suggestion was moving away from perl due to portability issues. I can
understand that, but what do you think about doing it in Java? Bash is nice,
but in some ways it suffers from the same portability issues that perl does.
Here are a few issues I have run into so far. The "swift" shell script we
use calls /bin/sh. Sometimes this is bash. On some linux systems it's
actually a shell called ash, which is similar but does not always
compatible. If you can assume there will always be a /bin/bash, there are
individual differences between bash versions. When I was testing changes to
usage stats, I ran into a really old version of bash which caused it to
fail. Then there are things like /dev/udp which may or may not be turned on
based on bash compilation options. Bash is nice, but it's a little limited
in what is can do by itself without relying on a lot of external
applications.. which may not be always be installed on the system or work as
expected.

The nice thing about doing it in Java is that you can bank on Java 1.5 or
later being there regardless of whatever system quirks you may run into.
Java also includes XML handling libraries and has nice regular expression
system which would make this a little easier to handle. It will also be
easier to manage as complexity increases in the future.

Shifting gears a bit.. what I would _really_ would like in the future is a
graphical interface on top of swiftconfig and swiftrun. I may be the only
one, but I haven't given up on this idea :-)

- Have one visual interface running locally which will let me do everything
- Easily and visually configure swift to submit jobs on a remote system
- Visually create a workflow which generates a swift script
- Select a swift script and be able to easily change parameters, queue,
maxtime, # nodes, etc on the fly
- See a visual progression of what is happening. Something like a swing
version of -tui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110202/4ace2bed/attachment.html>