[Swift-devel] swiftconfig

[wilde at mcs.anl.gov]

David, after reading the command line syntax you proposed, I realize this is going to take more thought and discussion.

What I was originally envisioning was a swiftconfig command that stored a set of templates for various sites, and adjusted them based on the individual user (eg the <workdirectory> and user-selected options (like local vs Globus vs ssh access; coasters vs non-coasters; and for coasters, a reduced set of options that could tailor the options similarly into a few common profiles).

Justin suggested in discussion today that we start by just building a manual catalog of a good set of examples that covers the local and grid systems that many of use use regularly, so that, at least, people can copy a documented example and adjust as needed.

This might a good exercise from which we could identify that patterns that a swiftconfig command could use to simplify the process.

I started trying to outline some of the patterns; we should match these up against profiles for the 8 or so systems that I mentioned in a message to this list last week (pads, fusion, ranger, etc).

Im starting to feel myself swayed by your suggests for an interactive interface, but I find it hard at the moment to see what that would look like. Possible a set of drop-downs or range selection boxes that adjust as the user narrows a site's profile in a fashion that mimics the outline below?

- Mike

1. Local immediate execution

    1.1 without coasters
    1.2 with coasters

2. Local scheduled execution

    2.1 without coasters

        2.1.1 PBS (eg: TeraPort, PADS, Fusion, many TG sites)
        2.1.2 SGE (eg: Ranger, godzilla, sisboombah)
        2.1.3 Condor (eg: Purdue TeraGrid Condor pool; HNL condor pool; UC Condor pool???)

    1.2 with coasters

        2.2.1 PBS (eg: TeraPort, PADS, Fusion, many TG sites
        2.2.2 SGE (eg: Ranger, godzilla, sisboombah)
        2.2.3 Condor (eg: Purdue TeraGrid Condor pool; HNL condor pool; UC Condor pool???)

3. ssh to remote sites

    3.1 to local immediate
    3.2 to coasters
        to coasters pbs, sge, and condor

4. GT2 to remote sites

    4.1 Non-Condor-G
        4.1.1 Plain
        4.1.2 Coasters
    4.2 Condor-G
        4.2.1 Plain
        4.2.2 Coasters

----- "David Kelly" <dk0966 at cs.ship.edu> wrote:

> Hello all,
> 
> I am working on a utility to modify configuration files called
> swiftconfig. This is still in the early stages, so there is a lot of
> room for changes and new ideas. I believe there is some overlap
> between this project and what some other students will be doing this
> summer, so if anyone would like to work with me on this, please feel
> free.
> 
> I envision swiftconfig as a simple text-based configuration program.
> It will be written in Perl and use the curses library for easier
> editing. It should hopefully make swift configuration a little easier
> and prevent silly mistakes like typos in xml which could keep swift
> from running. Everything that can be done within the editor should
> also be able to be done directly from the command line. This should
> make it easier to expand upon in the future. For example, a web or GUI
> based application could be written fairly quickly that would only need
> to call swiftconfig with the correct command line options.
> 
> There are three files swiftconfig can modify: tc.data, sites.xml, and
> auth.defaults.
> 
> The options for transformation mode include
> 
> -host # Host name
> -name # Translation name
> -path # Path to executable
> -profile # Profile arguments, defaults to null
> -tcfile # Location of tc file. If not specified, find tc.data based on
> location of swift
> -overwrite # If a duplicate is found, overwrite the old entry without
> prompting
> 
> Since platform and installation status are no longer used, they will
> default to INTEL32::LINUX and INSTALLED.
> Here is an example of swiftconfig in transformation mode.
> 
> $ swiftconfig -host localhost -name wc -path /usr/bin/wc
> 
> tc.data should then have the line:
> localhost wc /usr/bin/wc INSTALLED INTEL32::LINUX null
> 
> If there is already an entry with the name wc, it should prompt the
> user to answer yes/no if the user wants to overwrite it (unless
> -overwrite is given)
> 
> For sites.xml, swiftconfig should allow the user to use existing
> examples or specify their own. Here are the options:
> 
> -template # Use existing commented example for defaults (skynet,
> teraport, etc)
> -entry # Name of new entry (pool handle)
> -gridftp # Specify gridftp url
> -jobuniverse # Specify jobmanager universe
> -joburl # Specify jobmanager url
> -jobmajor # Specify jobmanager major value
> -jobminor # Specify jobmanager minor value
> -directory # Work directory
> -exprovider # Execution provider
> -exmanager # Execution job manager
> -exurl # Execution url
> -remove # Remove (comment out) an entry from sites.xml
> 
> So, for example suppose a user has the following entry in sites.xml by
> default:
> 
> <!--
> <pool handle="teraport" >
> <gridftp url="gsiftp:// tp-grid1.uchicago.edu " />
> <jobmanager universe="vanilla" url="
> tp-grid1.uchicago.edu/jobmanager-pbs " major="2" />
> <workdirectory >/home/tiberius/scratch/SWIFT-WORK</workdirectory>
> </pool>
> -->
> 
> The command:
> 
> $ swiftconfig -template teraport
> 
> Which would uncomment that from sites.xml as is. The user could also
> modify just a part of it:
> 
> $ swiftconfig -template teraport -directory /tmp
> 
> That should modify only the workdirectory and leave everything else
> the same.
> 
> To create your own config, use -entry instead of -template
> 
> $ swiftconfig -entry mynetwork -gridftp ftp.foo -exprovider gt4 (..
> and so on)
> 
> The final mode of swiftconfig is for auth.log in ssh configurations.
> 
> -auth # Set to auth mode
> -sshhost # Name of remote ssh host
> -sshmode # Either password or passphrase
> -sshuser # SSH username
> -sshpassword # SSH password
> -sshpassphrase # SSH passphrase
> -sshkey # Location of SSH key
> 
> Any other ideas or suggestions on how swiftconfig should work are
> welcome.
> 
> David
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory