[Swift-devel] 0.94 release note draft

[Michael Wilde]

David, this is a great list, and a great start to the Release Notes doc.  Before releasing, though, you need to take many of the items to the next level of detail.

If we dont want to change the Userguide prior to release, write sections in the release notes that will go into trunk Userguide.

Some notes, below.  Some are easy to fix.  For ones (eg MPI) where time may not permit, maybe say "Ask on the Swift list and watch the trunk doc for emerging details." ???

- Mike


----- Original Message -----
> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, March 1, 2013 7:06:05 AM
> Subject: Re: 0.94 release note draft
> 
> 
> Here is the list, based mostly on what I could find in the svn logs.
> 
> 
> 
> - To behavior of iterate has changed from 0.93 to 0.94. If you have
> scripts
> that use iterate, please read
> http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_iterate

Spell out what changed. Further, the userguide section on iterate needs more clarity.

Echo what changed, in the user guide (by adding a short note and example to the end marked "behavior prior to 0.94").

The example of i and j termination being different is important but needs to be clarified.

> - Walltimes are more strictly enforced by coasters.

Are now enforced, where before they were not.

> Previous to Swift
> version
> 0.94, if an application run with coasters would exceed its specified
> maxwalltime, it would be allowed to continue to execute. However, if
> this
> would cause the worker on which the application was running to exceed
> its
> maxwalltime, the queuing system would kill the worker. The resulting
> error
> message was not always very clear. Since version 0.94 coaster workers
> enforce
> the user-specified maxwalltime. If an application exceeds its
> maxwalltime,
> the coaster worker will not allow it to continue, but terminate it
> and report
> the error.

Need to explain the diff between maxtime and maxwalltime.  This is lacking in the userguide.

> 
> - Swift will now use camel case for functions, for example, @toInt
> instead of
> @toint. The previous naming convention will still work, but you may
> see
> deprecated warnings.

For example?

What do the messages look like? Exactly how is case-difference treated?
What were the old conventions and what are the new ones?

> - Associative arrays have been added. More details and examples can
> be found at
> http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_associative_arrays
 
> - Dynamic profiles. Many settings formerly only definable in
> sites.xml can now
> be set on a per-app basis. This can make things easier when running
> multiple apps that have different requirements for settings like
> processors
> per node and wallitme.
> http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_dynamic_profiles
> 
> 
> - Added a new ssh command line provider. Previously ssh support was
> done by
> creating a file called ~/.ssh/auth.defaults. The ssh command line
> provider
> is more flexible and doesn't require this step. ssh-cl allows you to
> use
> SSH agents. You can use ssh-cl by adding something like this to your
> sites.xml:
> 
> 
> <execution provider="coaster" url="my.host.uchicago.edu"
> jobmanager="ssh-cl:pbs"/>
> 
> 
> - Many fixes and improvements to improve the reliability and
> performance of
> coaster provider staging.
> 
> 
> - Added support for the Slurm scheduler

How to use? What are the parameter override/passing issues?

> 
> - Added support for the LSF scheduler

ditto

> 
> - Improvements to condor provider (non-shared jobtype and more
> flexibility
> to define what gets added to the submit script).

How to use?

> 
> - Fixes for the textual user interface (TUI). Adding the -tui option
> to the swift
> command line allows you to monitor progress in a curses based menu. A
> brief
> example of this can be found at
> http://www.ci.uchicago.edu/~davidk/modis.ogv.

Insert into README?

> 
> - Added the ability to call Java methods within swift using @java.
> For example:
> float f = @java("java.lang.Math", "sin", 0.5);
> http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_java
> 
> 
> - Added a hang checker that provides the user with more information
> about
> potential hangs

Much more explanation needed here.  WHat causes hangs?  Which are real, which may be "just slow"?

How to read the traceback?

> 
> - @strjoin function for joining strings.
> http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_strjoin

> 
> - If you have a requirement that a command get run on the worker node
> before
> processing any work, worker.pl will now execute commands stored in
> the
> environment variable $WORKER_INIT_CMD

How to format the var?

> 
> - Use $SWIFT_USERHOME to determine where to swift should create some
> of its
> required files. This defaults to $HOME, but this may cause problems
> in some
> situations where $HOME is not accessable on worker nodes.

Unclear. Still a problem, or thats when to use this?

> 
> - Experimental "wrapper staging" feature that delegates file staging
> to an
> external wrapper script.

DOnt mention this; no one can use it!

> 
> - Various improvements to the way that Swift runs MPI jobs.

Much much more needed here.

> 
> - Better OSG integration/support using GlideinWMS.

Ditto.


> 
> ----- Original Message -----
> 
> 
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Thursday, February 14, 2013 1:08:56 PM
> Subject: Fwd: 0.94 release note draft
> 
> 
> Some notes toward an 0.94 release notes document.
> 
> There's a longer list, I think in an IM chat transcript, that we need
> to incorporate.
> 
> Please send additional items to this thread for David to integrate.
> 
> Thanks,
> 
> - Mike
> 
> ----- Forwarded Message -----
> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Sent: Thursday, January 24, 2013 11:06:18 PM
> Subject: Re: 0.94 release note draft
> 
> Mike,
> 
> I just have the quick notes I took from our meeting. These combined
> with your emails are all the changes that I'm aware of at this
> point.
> 
> Iterate differences
> Walltime hard limit with coasters
> Associate arrays changes
> Tracebacker
> Coaster changes / parameters
> Slurm and LSF providers
> Condor provider changes
> ssh-cl
> TUI
> hang checker
> @functions (strjoin, and possibly others)
> Dynamic profiles
> Wrapper staging
> Pass-thru (PBS attributes)
> MPI support
> 
> 
> ----- Original Message -----
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Sent: Thursday, January 24, 2013 7:56:54 PM
> > Subject: 0.94 release note draft
> > 
> > 
> > Hi David,
> > 
> > I recall I sent you a few batches of line items to list in 0.94
> > release notes. Did you gather those somewhere where I can review
> > them? (Need them for a status report)
> > 
> > Thanks,
> > 
> > - Mike
> > 
> > 
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> > 
> > 
> 
>