[Swift-devel] Issues to resolve for the 0.96 config mechanism
Michael Wilde
wilde at anl.gov
Sun Jul 13 20:17:53 CDT 2014
I agree about the logical nature of the search path. But it was
designed so that most users would see a very simple view of
configuration. Its not really like a deep inheritance tree.
The concept was this:
> 1. The etc/swift.properties file included with the Swift distribution.
These would be basic things like localhost, and maybe a few simple sites
for generic cloud pools and sets of ad-hoc nodes.
Things that should enable you to just download Swift and try a few
things without thinking (or even knowing) about sites.
> 2. $SWIFT_SITE_CONF/swift.properties - used for defining site templates.
This was for site admins to make, e.g., Swift modules, so that when you
say "module load swift" you get a set of reasonable site definitions for
local clusters and queues.
> 3. $HOME/.swift/swift.properties
This would allow the user to set some preferences, like whether they
wanted Swift retry to be on or off by default.
> 4. swift.properties in your current directory.
This was seen as the *main* property file that a user would create,
alongside their .swift file, for a given scripting application. Its
where anything specific to running a given script would be set.
> 5. Any property file you point to with the command line argument
This was seen as a way to specify a remote configuration file, and was
expected to be seldom used by most users.
Regarding the tool, I agree totally, and there was indeed such a thing
provided in 0.95:
To verify what files are being read, and what values will be set, run:
-----
$ swift -listconfig
-----
Ive not checked its output, but both it, and the swift log, should list:
a) a set of all the properties files that were read in the run (or in
the current environment)
b) a hierarchical listing of all properties (including site attributes),
listed one attribute per line.
For each attribute, the output can append a simple [n] notation after
each attribute stating which of the N property files that attribute was
set from.
- Mike
On 7/13/14, 8:04 PM, Tim Armstrong wrote:
> My two cents: the five step search path seems logical and I can come
> up with scenarios where someone would want to have a particular
> setting at a particular level in the hierarchy.
>
> It reminds me though of some of the OOP debates about inheritance,
> where you have carefully thought out, deep, inheritance hierarchies,
> carefully engineered to reuse code wherever possible. The problem
> tends to be that as a developer reading a bit of code, you've got no
> idea which overridden method is being called, except maybe with the
> help of an IDE. There's not really a right answer, just that
> sometimes the smart complicated way is the best, and sometimes the
> dumb but simple way is the best.
>
> Maybe if having 4/5 levels of configuration is "the right thing" that
> users should do, all of the sample and tutorial properties files that
> users will copy could just explicitly include their predecessor? I.e.
> use the explicit mechanism but encourage the desired approach by
> convention.
>
> With the more complex inheritance mechanism, it seems like it is
> almost necessary to have a tool that shows all of the currently
> applied settings and where they come from.
>
> - Tim
>
>
> On Sun, Jul 13, 2014 at 6:37 PM, Mihael Hategan <hategan at mcs.anl.gov
> <mailto:hategan at mcs.anl.gov>> wrote:
>
> I know the feeling. I get it too. I see some new stuff and I was
> getting
> used to the old stuff, and I get this immediate anger about the new
> stuff and the feeling that it's stupid and I want the old stuff back.
>
> All I'm saying is give it some time. Try it out a bit, see how you
> would
> do something with it, etc. It's not set in stone. We have SVN and
> we can
> always go back.
>
> Ultimately the questions are: can I do what I was doing before,
> and how
> much effort does that require. Also what are the new things that I can
> do and could not do before, and how much better/worse is the overall
> process. I know what the previous thing could do, and you can still do
> that, just not in the exact same way.
>
> Some more inline...
>
> On Sun, 2014-07-13 at 17:03 -0500, Michael Wilde wrote:
> > On 7/13/14, 1:01 PM, Mihael Hategan wrote:
> > >> - the include mechanism is new. I think its nice and likely
> is very
> > >> >useful, but I wonder how it will interact with or supplement the
> > >> >property search path.
> > > We discussed this a few days ago. We had repeated issues with
> magically
> > > loaded files from strange locations that the users took a long
> time to
> > > find and fix. The solution that I saw in 0.95 was even more search
> > > locations, which I think was not right.
> > >
> > > So the philosophy in trunk is "either it stares at you or it isn't
> > > there". And includes do this in a way that doesn't sacrifice
> > > convenience.
> > >
> > I disagree with this. The properties search path in 0.95 was
> carefully
> > thought out, and based on a lot of experience with users, even
> it that
> > discussion did not take place on the list.
>
> You can write a config file that includes all the different config
> files
> and put it in swift/etc, if, as a user, that is what you think is
> best.
>
> >
> > Unfortunately the trunk commits over-wrote the 0.95
> documentation in the
> > trunk userguide, but the search path was like this:
> >
> > Location of swift.properties
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Swift searches for swift.properties files in multiple locations:
> >
> > 1. The etc/swift.properties file included with the Swift
> distribution.
> > 2. $SWIFT_SITE_CONF/swift.properties - used for defining site
> templates.
> > 3. $HOME/.swift/swift.properties
> > 4. swift.properties in your current directory.
> > 5. Any property file you point to with the command line argument
> > "-properties
> > <file>"
> >
> > Settings get read in this order. Definitions in the later files will
> > override
> > any previous definitions. For example, if you have
> execution.retries=10 in
> > $HOME/.swift/swift.properties, and execution.retries=0 in the
> > swift.properties
> > in your current directory, execution.retries will be set to 0.
>
> Yes. Well, I think this is bad. Imagine a user having to mentally go
> through this sequence in order to figure out what's being set and
> where.
> It's actually nearly impossible mentally. You have to go through the
> files and remember the order.
>
> The biggest problems we've had so far in terms of the configuration we
> had were with this. A file in the search path magically loaded without
> the user's knowledge. Even recently you and Yadu have experienced
> this,
> and were wondering where some configuration value comes from that was
> causing a run to break. And it took you a few hours and I had to get
> involved to sort it out. And It was all documented. See
> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1281
>
> So no, I do not think that this is a good idea.
>
> >
> > I thought this search path notion was logical, useful, and
> reflected the
> > needs of users and site admins. It was a refinement of what
> existed in
> > Swift going back many releases.
>
> It added extra steps to something that already had too many steps.
>
> That said, the new stuff doesn't say you can't do that. Just that you
> have to actually say it yourself that you want to do that. The
> reason we
> had multiple search paths is because we did not support includes,
> so we
> needed a mechanism to override only a few things. But you can simulate
> it with includes and customize it in whatever way you want and it
> would
> be obvious what's coming from where.
>
> The point is to give the user a simple and powerful tool to do
> what they
> need.
>
> >
> > In 0.95 it was tied to the use of run directories (which I hope
> has not
> > been removed from trunk!).
>
> No. It's still there. I like it. I only renamed runxxx.log to
> swift.log,
> since I did not se why everything had a name that did not include the
> run id except for the log. We can change this back if there is a
> reason.
>
> We do probably need to fix a few issues with it. It will break with
> concurrent starts of swift, and it will break on filesystems that
> don't
> list directories in some particular order.
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
--
Michael Wilde
Mathematics and Computer Science Computation Institute
Argonne National Laboratory The University of Chicago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140713/b5f4767e/attachment.html>
More information about the Swift-devel
mailing list