<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I agree about the logical nature of the search path. But it was
designed so that most users would see a very simple view of
configuration. Its not really like a deep inheritance tree.<br>
<br>
The concept was this:<br>
<br>
> 1. The etc/swift.properties file included with the Swift
distribution.<br>
<br>
These would be basic things like localhost, and maybe a few simple
sites for generic cloud pools and sets of ad-hoc nodes.<br>
Things that should enable you to just download Swift and try a few
things without thinking (or even knowing) about sites.<br>
<br>
> 2. $SWIFT_SITE_CONF/swift.properties - used for defining site
templates.<br>
<br>
This was for site admins to make, e.g., Swift modules, so that when
you say "module load swift" you get a set of reasonable site
definitions for local clusters and queues.<br>
<br>
> 3. $HOME/.swift/swift.properties<br>
<br>
This would allow the user to set some preferences, like whether they
wanted Swift retry to be on or off by default.<br>
<br>
> 4. swift.properties in your current directory.<br>
<br>
This was seen as the *main* property file that a user would create,
alongside their .swift file, for a given scripting application. Its
where anything specific to running a given script would be set.<br>
<br>
> 5. Any property file you point to with the command line
argument<br>
<br>
This was seen as a way to specify a remote configuration file, and
was expected to be seldom used by most users.<br>
<br>
Regarding the tool, I agree totally, and there was indeed such a
thing provided in 0.95:<br>
<br>
To verify what files are being read, and what values will be set,
run:<br>
-----<br>
$ swift -listconfig<br>
-----<br>
<br>
Ive not checked its output, but both it, and the swift log, should
list:<br>
<br>
a) a set of all the properties files that were read in the run (or
in the current environment)<br>
<br>
b) a hierarchical listing of all properties (including site
attributes), listed one attribute per line.<br>
For each attribute, the output can append a simple [n] notation
after each attribute stating which of the N property files that
attribute was set from.<br>
<br>
- Mike<br>
<br>
<br>
<div class="moz-cite-prefix">On 7/13/14, 8:04 PM, Tim Armstrong
wrote:<br>
</div>
<blockquote
cite="mid:CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>My two cents: the five step search path seems logical and I
can come up with scenarios where someone would want to have a
particular setting at a particular level in the hierarchy.<br>
</div>
<div><br>
It reminds me though of some of the OOP debates about
inheritance, where you have carefully thought out, deep,
inheritance hierarchies, carefully engineered to reuse code
wherever possible. The problem tends to be that as a
developer reading a bit of code, you've got no idea which
overridden method is being called, except maybe with the help
of an IDE. There's not really a right answer, just that
sometimes the smart complicated way is the best, and sometimes
the dumb but simple way is the best.<br>
</div>
<div><br>
</div>
Maybe if having 4/5 levels of configuration is "the right thing"
that users should do, all of the sample and tutorial properties
files that users will copy could just explicitly include their
predecessor? I.e. use the explicit mechanism but encourage the
desired approach by convention.<br>
<div>
<div><br>
With the more complex inheritance mechanism, it seems like
it is almost necessary to have a tool that shows all of the
currently applied settings and where they come from.<br>
<br>
</div>
<div>- Tim<br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Sun, Jul 13, 2014 at 6:37 PM, Mihael
Hategan <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:hategan@mcs.anl.gov" target="_blank">hategan@mcs.anl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">I know the
feeling. I get it too. I see some new stuff and I was
getting<br>
used to the old stuff, and I get this immediate anger about
the new<br>
stuff and the feeling that it's stupid and I want the old
stuff back.<br>
<br>
All I'm saying is give it some time. Try it out a bit, see
how you would<br>
do something with it, etc. It's not set in stone. We have
SVN and we can<br>
always go back.<br>
<br>
Ultimately the questions are: can I do what I was doing
before, and how<br>
much effort does that require. Also what are the new things
that I can<br>
do and could not do before, and how much better/worse is the
overall<br>
process. I know what the previous thing could do, and you
can still do<br>
that, just not in the exact same way.<br>
<br>
Some more inline...<br>
<div class=""><br>
On Sun, 2014-07-13 at 17:03 -0500, Michael Wilde wrote:<br>
> On 7/13/14, 1:01 PM, Mihael Hategan wrote:<br>
> >> - the include mechanism is new. I think
its nice and likely is very<br>
> >> >useful, but I wonder how it will
interact with or supplement the<br>
> >> >property search path.<br>
> > We discussed this a few days ago. We had
repeated issues with magically<br>
> > loaded files from strange locations that the
users took a long time to<br>
> > find and fix. The solution that I saw in 0.95
was even more search<br>
> > locations, which I think was not right.<br>
> ><br>
> > So the philosophy in trunk is "either it stares
at you or it isn't<br>
> > there". And includes do this in a way that
doesn't sacrifice<br>
> > convenience.<br>
> ><br>
> I disagree with this. The properties search path in
0.95 was carefully<br>
> thought out, and based on a lot of experience with
users, even it that<br>
> discussion did not take place on the list.<br>
<br>
</div>
You can write a config file that includes all the different
config files<br>
and put it in swift/etc, if, as a user, that is what you
think is best.<br>
<div class=""><br>
><br>
> Unfortunately the trunk commits over-wrote the 0.95
documentation in the<br>
> trunk userguide, but the search path was like this:<br>
><br>
> Location of swift.properties<br>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
> Swift searches for swift.properties files in multiple
locations:<br>
><br>
> 1. The etc/swift.properties file included with the
Swift distribution.<br>
> 2. $SWIFT_SITE_CONF/swift.properties - used for
defining site templates.<br>
> 3. $HOME/.swift/swift.properties<br>
> 4. swift.properties in your current directory.<br>
> 5. Any property file you point to with the command
line argument<br>
> "-properties<br>
> <file>"<br>
><br>
> Settings get read in this order. Definitions in the
later files will<br>
> override<br>
> any previous definitions. For example, if you have
execution.retries=10 in<br>
> $HOME/.swift/swift.properties, and
execution.retries=0 in the<br>
> swift.properties<br>
> in your current directory, execution.retries will be
set to 0.<br>
<br>
</div>
Yes. Well, I think this is bad. Imagine a user having to
mentally go<br>
through this sequence in order to figure out what's being
set and where.<br>
It's actually nearly impossible mentally. You have to go
through the<br>
files and remember the order.<br>
<br>
The biggest problems we've had so far in terms of the
configuration we<br>
had were with this. A file in the search path magically
loaded without<br>
the user's knowledge. Even recently you and Yadu have
experienced this,<br>
and were wondering where some configuration value comes from
that was<br>
causing a run to break. And it took you a few hours and I
had to get<br>
involved to sort it out. And It was all documented. See<br>
<a moz-do-not-send="true"
href="https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1281"
target="_blank">https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1281</a><br>
<br>
So no, I do not think that this is a good idea.<br>
<div class=""><br>
><br>
> I thought this search path notion was logical,
useful, and reflected the<br>
> needs of users and site admins. It was a refinement
of what existed in<br>
> Swift going back many releases.<br>
<br>
</div>
It added extra steps to something that already had too many
steps.<br>
<br>
That said, the new stuff doesn't say you can't do that. Just
that you<br>
have to actually say it yourself that you want to do that.
The reason we<br>
had multiple search paths is because we did not support
includes, so we<br>
needed a mechanism to override only a few things. But you
can simulate<br>
it with includes and customize it in whatever way you want
and it would<br>
be obvious what's coming from where.<br>
<br>
The point is to give the user a simple and powerful tool to
do what they<br>
need.<br>
<div class=""><br>
><br>
> In 0.95 it was tied to the use of run directories
(which I hope has not<br>
> been removed from trunk!).<br>
<br>
</div>
No. It's still there. I like it. I only renamed runxxx.log
to swift.log,<br>
since I did not se why everything had a name that did not
include the<br>
run id except for the log. We can change this back if there
is a reason.<br>
<br>
We do probably need to fix a few issues with it. It will
break with<br>
concurrent starts of swift, and it will break on filesystems
that don't<br>
list directories in some particular order.<br>
<div class="HOEnZb">
<div class="h5"><br>
<br>
_______________________________________________<br>
Swift-devel mailing list<br>
<a moz-do-not-send="true"
href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
<a moz-do-not-send="true"
href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel"
target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Wilde
Mathematics and Computer Science Computation Institute
Argonne National Laboratory The University of Chicago
</pre>
</body>
</html>