[Swift-devel] Swiftconfig merge / changes

Wed Feb 2 15:31:38 CST 2011

I was not aware that a java program could call another java programs 
main method.  By not aware I mean I do not know how.

On 2/2/11 3:26 PM, Mihael Hategan wrote:
> On Wed, 2011-02-02 at 13:22 -0800, Mihael Hategan wrote:
>> On Wed, 2011-02-02 at 00:04 -0600, jon.monette at gmail.com wrote:
>>> I agree switching the swiftrun/swiftconfig to using bash may not be
>>> the best idea due to the above reasons David has mentioned. However I
>>> do not believe switching over to using Java is the solution. In the
>>> end swiftrun is calling swift which is a shell script. Having a java
>>> process call a shell script which in turns starts java processes may
>>> not be the best plan of action.
>> That doesn't need to be so. A java program can invoke the main() method
>> of swift.
> In fact, this may very well turn out to fit into that swift shell model
> that Mike mentioned that would keep coaster workers (and other JVM
> persistent things) alive between swift runs.
>
>>>   May I suggest using python? Not much changes between python versions
>>> and it has very good xml libraries to work with. I do not believe
>>> there will be very many comparability issues in using python but in
>>> all programming there are always these issues. Python is also much
>>> easier to read in my humble opinion.
>>>
>>> ----- Reply message -----
>>> From: "David Kelly"<dk0966 at cs.ship.edu>
>>> Date: Tue, Feb 1, 2011 11:24 pm
>>> Subject: [Swift-devel] Swiftconfig merge / changes
>>> To: "Michael Wilde"<wilde at mcs.anl.gov>
>>> Cc: "swift-devel"<swift-devel at ci.uchicago.edu>
>>>
>>>
>>>
>>>
>>> On Tue, Feb 1, 2011 at 12:07 PM, Michael Wilde<wilde at mcs.anl.gov>
>>> wrote:
>>>
>>>          Also, Im very eager to reconcile and unify into one
>>>          coordinated plan the idea of merging the best aspects of
>>>          Justin's ad-hoc configuration methid, which he's documented on
>>>          the SWFT wiki (see eg CoasterCookbook etc) and your
>>>          swiftconfig/swiftrun mechanism.
>>>
>>>          I like the fact that Justin's is based on sh rather then perl
>>>          - that will make maintenance easier and reduce portability
>>>          issues. But I also want to bring back various concepts from
>>>          swiftconfig/run into this mechanism.  If we can agree on a
>>>          spec, perhaps that would be a good project for you to (resume)
>>>          work on?
>>>
>>> Here is what I see as the list of features we would need to add to the
>>> shell script in order to merge features:
>>>
>>>  From swiftconfig:
>>>
>>> - XML editor. It should have some basic knowledge of valid inputs (for
>>> example, when you're editing the execution provider setting, it will
>>> only allow you to select one that swift knows)
>>> - Editor for adding/removing/modifying apps to the tc file and
>>> verifying formatting
>>> - Using templates to generate new site configurations
>>> - Using templates to generate new templates
>>> - Importing existing configuration files into the template
>>> format/directory structure
>>> - Creating and managing ssh configurations for various hosts
>>>
>>>  From swiftrun:
>>>
>>> - A new swiftrun mechanism which reads from a template directory,
>>> generates the run.XXXX directory, links input data, and runs swift
>>> - The concept of site groups. For example, creating a group called
>>> "MCS-coasters" which combines the individual configurations of
>>> thrash-coasters, thwomp-coasters, etc.
>>> - Similar to above, application sets for handling different sets of
>>> apps
>>>
>>> It's possible we may not need or want all of these. I just wanted to
>>> point out what would be involved if we merged the main features as
>>> they currently exist in the swiftconfig/swiftrun utilities. I also
>>> have a list of suggestions for future improvements which include
>>> things like revision control for modifications and a tagging/search
>>> system for searching through previous swift runs.
>>>
>>> The suggestion was moving away from perl due to portability issues. I
>>> can understand that, but what do you think about doing it in Java?
>>> Bash is nice, but in some ways it suffers from the same portability
>>> issues that perl does. Here are a few issues I have run into so far.
>>> The "swift" shell script we use calls /bin/sh. Sometimes this is bash.
>>> On some linux systems it's actually a shell called ash, which is
>>> similar but does not always compatible. If you can assume there will
>>> always be a /bin/bash, there are individual differences between bash
>>> versions. When I was testing changes to usage stats, I ran into a
>>> really old version of bash which caused it to fail. Then there are
>>> things like /dev/udp which may or may not be turned on
>>> based on bash compilation options. Bash is nice, but it's a little
>>> limited in what is can do by itself without relying on a lot of
>>> external applications.. which may not be always be installed on the
>>> system or work as expected.
>>>
>>> The nice thing about doing it in Java is that you can bank on Java 1.5
>>> or later being there regardless of whatever system quirks you may run
>>> into. Java also includes XML handling libraries and has nice regular
>>> expression system which would make this a little easier to handle. It
>>> will also be easier to manage as complexity increases in the future.
>>>
>>> Shifting gears a bit.. what I would _really_ would like in the future
>>> is a graphical interface on top of swiftconfig and swiftrun. I may be
>>> the only one, but I haven't given up on this idea :-)
>>>
>>> - Have one visual interface running locally which will let me do
>>> everything
>>> - Easily and visually configure swift to submit jobs on a remote
>>> system
>>> - Visually create a workflow which generates a swift script
>>> - Select a swift script and be able to easily change parameters,
>>> queue, maxtime, # nodes, etc on the fly
>>> - See a visual progression of what is happening. Something like a
>>> swing version of -tui
>>>
>>>
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>