[Swift-devel] 0.94 release note draft

Fri Mar 1 15:32:20 CST 2013

David,

Have mercy on me having lost track of the situation. Maybe tomorrow, most definitely next week I will start to increase the size of some of my swift runs till I hit the thousands of jobs.
Some will be very heavy in I/O and computations (and pipeline complexity).

I have taken today off to recover  and be ready.

What do you suggest should be my approach to this problem? 
I am currently trying to develop pipelines with 

SWIFT_HOME=/soft/swift/0.94-2012.1102

or 

#SWIFT_HOME=/home/davidk/swift-trunk/cog/modules/swift/dist/swift-svn

then move them to fast and see where it breaks.

Should i change the module I am using? I need to learn how to make the installation work and get over my hatred for java...

All the pipelines I will be trying are expected eventually to reach sustained runs of 10K+ jobs in parallel with workflows as deep as 20 stages (including steps that increase the jobs by one or two orders of magnitude with a total file load of a few tens of TBs)

Do you think that it is sensible?

We will then mover to other machines and hopefully test fusion and other approaches.

Lorenzo

On Mar 1, 2013, at 3:13 PM, David Kelly wrote:

> I think it is there. It looks like your memory fixes were committed on 2-2, and the current 0.94 branch is based on a snapshot of trunk from 2-15. 
> 
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Michael Wilde" <wilde at mcs.anl.gov>, "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, March 1, 2013 12:36:03 PM
> Subject: Re: [Swift-devel] 0.94 release note draft
> 
> So I don't think I committed the memory leak fixes to 0.94 and I think
> they should be there.
> 
> Mihael
> 
> On Fri, 2013-03-01 at 07:06 -0600, David Kelly wrote:
> > Here is the list, based mostly on what I could find in the svn logs. 
> > 
> > 
> > 
> > - To behavior of iterate has changed from 0.93 to 0.94. If you have scripts 
> > that use iterate, please read 
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_iterate 
> > 
> > 
> > - Walltimes are more strictly enforced by coasters. Previous to Swift version 
> > 0.94, if an application run with coasters would exceed its specified 
> > maxwalltime, it would be allowed to continue to execute. However, if this 
> > would cause the worker on which the application was running to exceed its 
> > maxwalltime, the queuing system would kill the worker. The resulting error 
> > message was not always very clear. Since version 0.94 coaster workers enforce 
> > the user-specified maxwalltime. If an application exceeds its maxwalltime, 
> > the coaster worker will not allow it to continue, but terminate it and report 
> > the error. 
> > 
> > 
> > - Swift will now use camel case for functions, for example, @toInt instead of 
> > @toint. The previous naming convention will still work, but you may see 
> > deprecated warnings. 
> > 
> > 
> > - Associative arrays have been added. More details and examples can be found at 
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_associative_arrays 
> > 
> > 
> > - Dynamic profiles. Many settings formerly only definable in sites.xml can now 
> > be set on a per-app basis. This can make things easier when running 
> > multiple apps that have different requirements for settings like processors 
> > per node and wallitme. 
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_dynamic_profiles 
> > 
> > 
> > - Added a new ssh command line provider. Previously ssh support was done by 
> > creating a file called ~/.ssh/auth.defaults. The ssh command line provider 
> > is more flexible and doesn't require this step. ssh-cl allows you to use 
> > SSH agents. You can use ssh-cl by adding something like this to your sites.xml: 
> > 
> > 
> > <execution provider="coaster" url="my.host.uchicago.edu" jobmanager="ssh-cl:pbs"/> 
> > 
> > 
> > - Many fixes and improvements to improve the reliability and performance of 
> > coaster provider staging. 
> > 
> > 
> > - Added support for the Slurm scheduler 
> > 
> > 
> > - Added support for the LSF scheduler 
> > 
> > 
> > - Improvements to condor provider (non-shared jobtype and more flexibility 
> > to define what gets added to the submit script). 
> > 
> > 
> > - Fixes for the textual user interface (TUI). Adding the -tui option to the swift 
> > command line allows you to monitor progress in a curses based menu. A brief 
> > example of this can be found at http://www.ci.uchicago.edu/~davidk/modis.ogv. 
> > 
> > 
> > - Added the ability to call Java methods within swift using @java. For example: 
> > float f = @java("java.lang.Math", "sin", 0.5); 
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_java 
> > 
> > 
> > - Added a hang checker that provides the user with more information about 
> > potential hangs 
> > 
> > 
> > - @strjoin function for joining strings. 
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_strjoin 
> > 
> > 
> > - If you have a requirement that a command get run on the worker node before 
> > processing any work, worker.pl will now execute commands stored in the 
> > environment variable $WORKER_INIT_CMD 
> > 
> > 
> > - Use $SWIFT_USERHOME to determine where to swift should create some of its 
> > required files. This defaults to $HOME, but this may cause problems in some 
> > situations where $HOME is not accessable on worker nodes. 
> > 
> > 
> > - Experimental "wrapper staging" feature that delegates file staging to an 
> > external wrapper script. 
> > 
> > 
> > - Various improvements to the way that Swift runs MPI jobs. 
> > 
> > 
> > - Better OSG integration/support using GlideinWMS. 
> > 
> > 
> > ----- Original Message -----
> > 
> > 
> > From: "Michael Wilde" <wilde at mcs.anl.gov> 
> > To: "David Kelly" <davidk at ci.uchicago.edu> 
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu> 
> > Sent: Thursday, February 14, 2013 1:08:56 PM 
> > Subject: Fwd: 0.94 release note draft 
> > 
> > 
> > Some notes toward an 0.94 release notes document. 
> > 
> > There's a longer list, I think in an IM chat transcript, that we need to incorporate. 
> > 
> > Please send additional items to this thread for David to integrate. 
> > 
> > Thanks, 
> > 
> > - Mike 
> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130301/bdbec208/attachment.html>