[Swift-devel] 0.94 release note draft

[David Kelly]

Lorenzo, 

That seems reasonable to me. I haven't used the faster branch myself much at this point, so it's probably best to continue talking with Mihael and Mike to come up with a good strategy for how to approach those issues. But in terms of the swift versions/modules, I have put together the three latest builds for you at: 

/home/davidk/swift-0.94-03012013/ cog/modules/swift/dist/swift-svn 
/home/davidk/swift-faster-03012013/cog/modules/swift/dist/swift-svn 
/home/davidk/swift-trunk-03012013/cog/modules/swift/dist/swift-svn 

It's probably better to use these rather than ~davidk/swift-trunk, because I tend to use that for testing. Hope this helps. 

David 

----- Original Message -----

> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Mihael Hategan" <hategan at mcs.anl.gov>, "Swift Devel"
> <swift-devel at ci.uchicago.edu>
> Sent: Friday, March 1, 2013 3:32:20 PM
> Subject: Re: [Swift-devel] 0.94 release note draft

> David,

> Have mercy on me having lost track of the situation. Maybe tomorrow,
> most definitely next week I will start to increase the size of some
> of my swift runs till I hit the thousands of jobs.
> Some will be very heavy in I/O and computations (and pipeline
> complexity).

> I have taken today off to recover and be ready.

> What do you suggest should be my approach to this problem?
> I am currently trying to develop pipelines with

> SWIFT_HOME=/soft/swift/0.94-2012.1102

> or

> #SWIFT_HOME=/home/davidk/swift-trunk/cog/modules/swift/dist/swift-svn

> then move them to fast and see where it breaks.

> Should i change the module I am using? I need to learn how to make
> the installation work and get over my hatred for java...

> All the pipelines I will be trying are expected eventually to reach
> sustained runs of 10K+ jobs in parallel with workflows as deep as 20
> stages (including steps that increase the jobs by one or two orders
> of magnitude with a total file load of a few tens of TBs)

> Do you think that it is sensible?

> We will then mover to other machines and hopefully test fusion and
> other approaches.

> Lorenzo

> On Mar 1, 2013, at 3:13 PM, David Kelly wrote:

> > I think it is there. It looks like your memory fixes were committed
> > on 2-2, and the current 0.94 branch is based on a snapshot of trunk
> > from 2-15.
> 

> > ----- Original Message -----
> 

> > > From: "Mihael Hategan" < hategan at mcs.anl.gov >
> > 
> 
> > > To: "David Kelly" < davidk at ci.uchicago.edu >
> > 
> 
> > > Cc: "Michael Wilde" < wilde at mcs.anl.gov >, "Swift Devel" <
> > > swift-devel at ci.uchicago.edu >
> > 
> 
> > > Sent: Friday, March 1, 2013 12:36:03 PM
> > 
> 
> > > Subject: Re: [Swift-devel] 0.94 release note draft
> > 
> 

> > > So I don't think I committed the memory leak fixes to 0.94 and I
> > > think
> > 
> 
> > > they should be there.
> > 
> 

> > > Mihael
> > 
> 

> > > On Fri, 2013-03-01 at 07:06 -0600, David Kelly wrote:
> > 
> 
> > > > Here is the list, based mostly on what I could find in the svn
> > > > logs.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - To behavior of iterate has changed from 0.93 to 0.94. If you
> > > > have
> > > > scripts
> > 
> 
> > > > that use iterate, please read
> > 
> 
> > > > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_iterate
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Walltimes are more strictly enforced by coasters. Previous to
> > > > Swift version
> > 
> 
> > > > 0.94, if an application run with coasters would exceed its
> > > > specified
> > 
> 
> > > > maxwalltime, it would be allowed to continue to execute.
> > > > However,
> > > > if this
> > 
> 
> > > > would cause the worker on which the application was running to
> > > > exceed its
> > 
> 
> > > > maxwalltime, the queuing system would kill the worker. The
> > > > resulting error
> > 
> 
> > > > message was not always very clear. Since version 0.94 coaster
> > > > workers enforce
> > 
> 
> > > > the user-specified maxwalltime. If an application exceeds its
> > > > maxwalltime,
> > 
> 
> > > > the coaster worker will not allow it to continue, but terminate
> > > > it
> > > > and report
> > 
> 
> > > > the error.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Swift will now use camel case for functions, for example,
> > > > @toInt
> > > > instead of
> > 
> 
> > > > @toint. The previous naming convention will still work, but you
> > > > may
> > > > see
> > 
> 
> > > > deprecated warnings.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Associative arrays have been added. More details and examples
> > > > can
> > > > be found at
> > 
> 
> > > > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_associative_arrays
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Dynamic profiles. Many settings formerly only definable in
> > > > sites.xml can now
> > 
> 
> > > > be set on a per-app basis. This can make things easier when
> > > > running
> > 
> 
> > > > multiple apps that have different requirements for settings
> > > > like
> > > > processors
> > 
> 
> > > > per node and wallitme.
> > 
> 
> > > > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_dynamic_profiles
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Added a new ssh command line provider. Previously ssh support
> > > > was
> > > > done by
> > 
> 
> > > > creating a file called ~/.ssh/auth.defaults. The ssh command
> > > > line
> > > > provider
> > 
> 
> > > > is more flexible and doesn't require this step. ssh-cl allows
> > > > you
> > > > to use
> > 
> 
> > > > SSH agents. You can use ssh-cl by adding something like this to
> > > > your sites.xml:
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > <execution provider="coaster" url=" my.host.uchicago.edu "
> > > > jobmanager="ssh-cl:pbs"/>
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Many fixes and improvements to improve the reliability and
> > > > performance of
> > 
> 
> > > > coaster provider staging.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Added support for the Slurm scheduler
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Added support for the LSF scheduler
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Improvements to condor provider (non-shared jobtype and more
> > > > flexibility
> > 
> 
> > > > to define what gets added to the submit script).
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Fixes for the textual user interface (TUI). Adding the -tui
> > > > option to the swift
> > 
> 
> > > > command line allows you to monitor progress in a curses based
> > > > menu.
> > > > A brief
> > 
> 
> > > > example of this can be found at
> > > > http://www.ci.uchicago.edu/~davidk/modis.ogv .
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Added the ability to call Java methods within swift using
> > > > @java.
> > > > For example:
> > 
> 
> > > > float f = @java("java.lang.Math", "sin", 0.5);
> > 
> 
> > > > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_java
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Added a hang checker that provides the user with more
> > > > information
> > > > about
> > 
> 
> > > > potential hangs
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - @strjoin function for joining strings.
> > 
> 
> > > > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_strjoin
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - If you have a requirement that a command get run on the
> > > > worker
> > > > node before
> > 
> 
> > > > processing any work, worker.pl will now execute commands stored
> > > > in
> > > > the
> > 
> 
> > > > environment variable $WORKER_INIT_CMD
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Use $SWIFT_USERHOME to determine where to swift should create
> > > > some of its
> > 
> 
> > > > required files. This defaults to $HOME, but this may cause
> > > > problems
> > > > in some
> > 
> 
> > > > situations where $HOME is not accessable on worker nodes.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Experimental "wrapper staging" feature that delegates file
> > > > staging to an
> > 
> 
> > > > external wrapper script.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Various improvements to the way that Swift runs MPI jobs.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Better OSG integration/support using GlideinWMS.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > ----- Original Message -----
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > From: "Michael Wilde" < wilde at mcs.anl.gov >
> > 
> 
> > > > To: "David Kelly" < davidk at ci.uchicago.edu >
> > 
> 
> > > > Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >
> > 
> 
> > > > Sent: Thursday, February 14, 2013 1:08:56 PM
> > 
> 
> > > > Subject: Fwd: 0.94 release note draft
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Some notes toward an 0.94 release notes document.
> > 
> 
> > > >
> > 
> 
> > > > There's a longer list, I think in an IM chat transcript, that
> > > > we
> > > > need to incorporate.
> > 
> 
> > > >
> > 
> 
> > > > Please send additional items to this thread for David to
> > > > integrate.
> > 
> 
> > > >
> > 
> 
> > > > Thanks,
> > 
> 
> > > >
> > 
> 
> > > > - Mike
> > 
> 

> > _______________________________________________
> 
> > Swift-devel mailing list
> 
> > Swift-devel at ci.uchicago.edu
> 
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130301/ae822fc8/attachment.html>