[Swift-devel] First tests with swift faster

[Michael Wilde]

> > Should we consider a simpler name/value format and eliminate XML
> > here? A site could pretty much be defined by a single level set of
> > name=value pairs.
> 
> Possibly. We do have to describe hierarchical information
> (site->(properties->sub-properties, apps->properties)), so the
> questions
> is whether name=value pairs will do the trick for that.

gensites substitutes named values into template sites.xml files.

We need to refine how the templates are located, and where site defaults are set (ie, whether there is a hierarchy of settings as well as a hierarchical structure within settings.

The sites.xml approach we have today is OK, as long as the user typically needs to only specify a few values into the settings.

Whats mostly broken today is attribute names that have gotten messy and unclear over time and add noise to the user's understanding (eg: globus profile vs karajan profile; maxtime vs maxwalltime).

So we can stay with sites.xml and gensites, but cleaning up the parsing may be a good time to clean up the attribute names.
 
> > Basically make the input to gensites the actual format instead of
> > requiring another layer?
> 
> Iff the input to gensites can fully describe the site "space", then
> yes.
> If not, then probably not.

I suggest we revisit gensites as a group. Till now David has been using it but its not been socialized among the group nor properly exposed to users yet.

Also, Lorenzo has a nice shell mechanism which ascii-prompts the user for run parameters; we might want to try to generalize it.

This all ties back to the "swiftrun" command, and gensites and swiftrun and any changes to the config mechanism need to be designed together.

> > 
> > I should also mention that what goes on *inside* the Java code is
> > similarly complex.  Understanding how attributes go from sites.xml
> > and
> > swift.properties through the various config layers, especially with
> > Coasters involved, has been a barrier to many developers, debugging
> > efforts, and improvements.  Im not sure the two are that closely
> > related, but while we're working in the neighborhood, might want to
> > consider some improvement (and documentation) as well.  I think
> > Justin
> > took a first crack at documenting the config conventions, but Im
> > not
> > sure where that went.
> 
> I think we're trying to abstract things away. In the same spirit that
> you don't really think about how CPU registers are assigned when you
> write some C program, you also shouldn't think about how exactly
> coasters are starring jobs.

Im not sure I understand. Im not questioning abstraction. Im pointing out that the manner in which parameters make it from outside the code to inside the code has been a blocker to developers.  Perhaps the current abstractions are OK, perhaps they could be improved.  But they have proven difficulty to understand and use. Often just getting to the right sites parameter values in the Java code has been more difficult than dealing with the given algorithmic problem the developer was trying to solve (e.g., adding new parameters for new sites like Crays, SGE, and SLURM).

Maybe that just needs documentation, or maybe it needs cleanup.

But if, for example, we change to JSON for the external sites parameter spec, that might be reflected in a simpler and more clear abstraction for passing parameters inside the code.  Similarly XMS and DOM, perhaps.

- Mike