[Swift-devel] 0.9 release notes draft

[Michael Wilde]

Thats a nice package of work. Kudos to all involved!

I think documenting some thing sabout coasters is important, primarily 
how jobs are sent to workers based on time (and that there is a default 
time of 10 minutes if time is not specified). I found this very 
confusing - not because its complicated, but because the current text on 
this topic was incomplete and not clear enough).

Mihael, can you do a read through the user guid and parameter 
descriptions on coasters, and either write whats needed or list what 
needs to be written later?

Other minor things from a quick read:

 > *** when replication is enabled, swift will locally kill jobs that have
 >     run for twice their specified walltime

what does "locally kill" mean?

 > *** Recompilation will happen if a .kml file was compiled with a 
different

This was an occasional problem: does this close the last know glitch in 
avoiding recompilation, or are there any other cases where a user could 
get tripped up on this? I.e., ideally should the user not need to know 
that Swift applies this heuristic?

- Mike


On 4/24/09 7:50 AM, Ben Clifford wrote:
> Below are the draft release notes for 0.9. They need more formatting, but 
> all the text is there.
> 
> I have said little about coasters other than document two new 
> configuration parameters and 'substantial ongoing development work'. If 
> there are any specific additional points to be made there, comment here.
> 
> ======
> 
> 
> User interface
> ==============
> 
> *** Added console text interface to provide live information about swift
>     runs, which can be enabled with the -tui commandline parameter
> 
> *** when replication is enabled, swift will locally kill jobs that have
>     run for twice their specified walltime
> 
> Execution modes
> ===============
> 
> *** Support for Condor-G submissions, by setting a job specification attribute
>     of "grid" and specifying a gridResource attribute containing the string
>     to be placed into the Condor-G grid_resource classad.
> 
> *** Support for submissions to a local condor pool
> 
> *** Coasters: substantial ongoing development work
> 
> Configuration parameters
> ========================
> 
> *** Environment variable SWIFT_EXTRA_INFO, when set in an environment profile,
>     is executed with the result being recorded in wrapper logs. This allows
>     arbitrary information about the remote environment to be gathered and
>     returned to the submit side.
> 
> *** New configuration option wrapper.invocation.mode, specifiable either
>     globally in the configuration file or per-site as a profile key,
>     that configures whether wrapper script invocations are made with an
>     absolute path (as was the behaviour in Swift 0.8) or with a relative
>     path (as was behaviour in previous versions of Swift).
> 
> ***     coasterWorkerMaxwalltime - a coaster parameter to explicitly set worker maxwalltime, overriding the default computation based on job walltimes. this should be useful when it is desirable to specify worker parameters based on the known properties of the target queue rather than on properties of the jobs to be executed inside coasters.
> 
> ***     coasterInternalIP parameter that allows the IP address used by coaster workers to communicate to the coaster head job to be set explicitly. this can be used when running on a cluster which an internal network which cannot access the IP address that the head node picks
> 
> *** TODO maxheapsize setting on commandline...
> 
> *** property ticker.disable to disable runtime ticker display
> 
> *** strsplit function which will split the input string based on separators that match the given pattern and return a string array.
> 
> New commands
> ============
> 
> *** The log-processing code, primarily exposed as the swift-plot-log command,
>     has been merged into the main Swift release, rather than being a separate
>     download.
> 
> *** There is a new command swift-osg-ress-site-catalog which will generate
>     a site catalog based on data gathered from OSG's ReSS information
>     system. This can be used by OSG users to easily generate a large number
>     of site catalog entries for sites that are likely to work.
> 
> Language changes
> ================
> 
> *** Procedure invocations can be made in any expression context now, rather
>     than only directly in an assignment.
> 
> *** Mappings can now be made in any declaration, whether it has an assignment
>     or not. Previous a procedure call assignment and a mapping could not be
>     made in the same statement.
> 
> *** Handling of [*] and . has changed. [*] has been broken for several
>     Swift releases, but now has some approximation of its former behaviour,
>     in a cleaner implementation.
>     [*] is now an identity operation, that is array[*] == array.
>     The structure access operator . can now take an array on the left
>     hand side. In situations where a[*].img was used, it is permissible
>     to continue to write a[*].img, or to write a.img - both of these will
>     return an array of img elements.
> 
> *** Tighter type checking on app blocks. Primitive types and arrays of
>     primitive types are permitted. Other types are prohibited.
> 
> *** Arrays of primitive types can be passed to app blocks, and will be
>     expanded into multiple command-line parameters, one parameter per
>     array element.
> 
> *** != operator had been broken forever. It works now.
> 
> *** output of trace() is changed for non-primitive datasets:
> 
>     * Array handling:
>     Previous to this, trace would show the Java object representation of a (Future)PairIterator for arrays, which is fairly useless for a user. This commit makes trace show more about the array, and only emit the trace when the array is closed.
> 
>     * Other datasets:
>     Trace will wait for those datasets to be closed, and emit their default
>     string representation (including the variable name and path used in
>     SwiftScript)
> 
> *** loop condition in iterate statements can now refer to variables declared within the iteration body (this was bug 177)
> 
> ***    Mappings can now be made in any declaration, whether it has an assignment or not.
> 
> 
> 
> Deprecations and removal of old functionality
> =============================================
> 
> *** Removed support for .dtm file extension which was deprecated in Swift 0.4
> 
> Internal changes
> ================
> 
> *** The wrapper.sh and seq.sh scripts that are deployed to remote sites to
>     help with execution have been renamed to more Swift specific names, to
>     avoid collision with user-supplied files of the same name. The new names
>     are _swiftwrap and _swiftseq
> 
> *** Recompilation will happen if a .kml file was compiled with a different
>     version of Swift to the version being invoked. This is in addition to the
>     existing behaviour where a .swift file will be recompiled if it is newer
>     than the corresponding .kml file.
> 
> *** Added a throttling parallelFor and changed the swift foreach underlying
>     implementation to it. The throttling parallelFor limits the number of
>     concurrent iterations allowed, which should allow swift to scale better
>     in certain cases. This is controlled by the configuration property
>     foreach.max.threads
>