[Swift-devel] Planning next set of Swift features and releases

[Ben Clifford]

I would argue fairly strongly against doing anything particularly 
revolutionary before a 1.0 release - the time for developing significant 
1.0 new features, if 1.0 is to be in August, is pretty much over (if you 
regard the development period for that being the past 3 years); experience 
has shown that it takes many many months for there to be enough 
developer/user iterations to get stuff working well.

Pay attention especially to the special significance of calling a release 
1.0 - nothing in there should be a "development feature" that has only 
recently been made, and perhaps little development time should be spent 
after the son-of-0.9 release on working on new stuff vs fixing existing 
stuff.

> Ian challenged me to narrow it to 10 features; I came close but have not yet
> given it the thought it needs, looked through bugzilla, etc. Then I backslid,
> but to pick 10, we need to consider more.

I think picking 10 things when those things can vary wildly in scope, is 
not a good thing to do. 

The below commentary is wrt what can be done for an august 1.0 release, 
not whether they are worth doing in general in the longer term.

> Clean up mapper model.

definitely not. far too much scope for breakage, and definitely breaking 
of everyones understanding of how mappers work now.

> Structure assignment and related issues
> 
>   - remove limitations on this
>   - maybe some similar language limits to address

needs more description. but there are a relatively small things that can 
be implemented that lok more like bugs in implementation / 
not-implemented-yet that are ok to do. anything that is significantly 
changing existing semantics should not happen.

> Good run time error messages with source code line numbers
> 
>   - summary of errors at end of output is ineffective
>   - review of messages in log that should go to output
>   - review of message content in log

yes.

> Coaster polishing and testing/robustness
> 
>   - address scheduler anomalies
>   - 3 modes of provisioning
> 


> Condor-G support

its there in 0.9rc2 but needs more testing and debugging.

> tc.data and PATH extensions

neither of the two subbullets talk about PATH

>  - rename tc.data (e.g.: applist.txt)?

no. changing one name to another doesn't help people, and breaks both 
existing users and documentation.

>  -  clean up its format, make simpler to add apps.

yes. this is a usability-killer.

> 
> app() syntax review
> 
>  - trailing ";" confusing
>  - quoting review and improvements if needed
>  - multi-line

If someone can come up with a compelling new format for app syntax then it 
should be straightforward to implement it. But coming up with that format 
is the hard thing. I don't think that lopping off a semicolon makes it 
particularly less or more intuitive compared to the rest of the app 
syntax.

> Auto app install

singel-executable staging is< i think, easily achieveable. anything more 
complex than that is not achievable for a 1.0 release. That does not 
preclude ongoing ADEM (or related to ADEM) development as a separate 
component (much as Falkon is or the log-processing stuff was)

> Make sites & tc config easy/instant for osg, tg, local clusters

that is too wishywashy a goal.

for osg, I am pleased with the present site generation stuff. my main 
concern is that it does not generate tc.data entries for /bin/sh, which is 
how self-deployed apps are made at the moment. But this might be addressed 
either in the single-app-deployment or tc.data points above.

>  - productive ADEM

this should be orthogonal to Swift releases.

> namespace::name:version for function/app identifiers

no. the implementation of this runs too deep throughout the code, and 
there is no real consensus at the moment on how these should behave.

> Global variables

yes. I think pretty easy and non-disruptive

> Swift command
> 
>  - multi-site selection

By that I take it to mean a commandline parameter to specifically restrict 
the set of sites to run on. in which case, yes, should be easy.

>  - run management (maybe a separate command)

This needs breaking up into specific features, many of which the answer 
will be yes to.

> Scaling
> 
>  - Efficient handling of 1M and more jobs per wf
>  - what else needed here?

At least some of this is karajan threading work, which pretty much means 
hategan has to do it. I don't pretend to understand how easy or hard it 
is, but I would be very wary about introducing major changes to the 
threading system there.

> Library / #include feature

By library here I take it youmean including swift code, not linking to 
other non-swift libraries. in which case, yes, I think a simple include 
mechanism can go in - you already have prototypes that show the semantics.

> Provenance
> 
>  - make it end-user useful and ready
>  - easy to get perf numbers
>  - easy to associate swift parameters with files and runs

yes.

(the rest of this paragraph is a rant) I am fairly happy with the ongoing 
development of this at the moment for PC3. However, there has been a 
consistent lack of feedback for the provenance work. I attribute no blame 
on any person for this, but it is the case that no on aside from Luiz has 
spent a significant amount of time working with the provenance code over 
the 16 months that it has been in the SVN. People throw in plenty of 
feature requests, some of which already exist, and then make no use of 
them. If other people in the group do not test out features in the 
provenance code properly (i.e. more than spending one hour every month) 
then people cannot complain when the features do not work how they want. 
The same applies to the log processing graphs - people need to come up 
with concrete technical criticisms rather than vague "please make it 
better"

> More logging / status reporting enhancement

>  - enhance format and info content of log tools report

see the above rant.

> 
> Review of iteration and related flow of control issues

>  - why iterate seems hard to use
>  - for, do, while?
>  - functional equivalents

Major changes in the language should not happen before 1.0. So no.

Ongoing review of the semantics and syntax is a good thing to do, distinct 
from 1.0. However, I think you need to have a better understanding of what 
the semantics are now before starting to suggest changes.

> Built-in and/or intrinsic functions

>  - remove @

yes. pretty easy, I think.

>  - additional string functions, perhaps other related builtins
>  - (time, date, sizeof, ...)

I don't think we should go about adding large numbers of builtins without 
clear use for them. So i think each proposed builtin shoudl be proposed on 
its individual merits. so there is no yes/no answer for this poitn.

>  - Built-in function extensibility

what does that mean?

>  - general external function mechanism - shell out? lib path?

no for 1.0 - mechanisms already exist for this, although they are 
syntactically ugly.

As this list is primarily about 1.0 features, I do not comment on your 
Long-term section which appeared here, other than to say they are all 
definitely post 1.0

--