[Swift-devel] 0.94 release note draft
Lorenzo Pesce
lpesce at uchicago.edu
Fri Mar 1 19:08:43 CST 2013
Thanks a million. I will proceed with testing and try to report problems timely and clearly.
Things are looking good.
On Mar 1, 2013, at 4:22 PM, David Kelly wrote:
> Lorenzo,
>
> That seems reasonable to me. I haven't used the faster branch myself much at this point, so it's probably best to continue talking with Mihael and Mike to come up with a good strategy for how to approach those issues. But in terms of the swift versions/modules, I have put together the three latest builds for you at:
>
> /home/davidk/swift-0.94-03012013/cog/modules/swift/dist/swift-svn
> /home/davidk/swift-faster-03012013/cog/modules/swift/dist/swift-svn
> /home/davidk/swift-trunk-03012013/cog/modules/swift/dist/swift-svn
>
> It's probably better to use these rather than ~davidk/swift-trunk, because I tend to use that for testing. Hope this helps.
>
> David
>
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Mihael Hategan" <hategan at mcs.anl.gov>, "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, March 1, 2013 3:32:20 PM
> Subject: Re: [Swift-devel] 0.94 release note draft
>
> David,
>
> Have mercy on me having lost track of the situation. Maybe tomorrow, most definitely next week I will start to increase the size of some of my swift runs till I hit the thousands of jobs.
> Some will be very heavy in I/O and computations (and pipeline complexity).
>
> I have taken today off to recover and be ready.
>
> What do you suggest should be my approach to this problem?
> I am currently trying to develop pipelines with
>
> SWIFT_HOME=/soft/swift/0.94-2012.1102
>
> or
>
> #SWIFT_HOME=/home/davidk/swift-trunk/cog/modules/swift/dist/swift-svn
>
> then move them to fast and see where it breaks.
>
> Should i change the module I am using? I need to learn how to make the installation work and get over my hatred for java...
>
> All the pipelines I will be trying are expected eventually to reach sustained runs of 10K+ jobs in parallel with workflows as deep as 20 stages (including steps that increase the jobs by one or two orders of magnitude with a total file load of a few tens of TBs)
>
> Do you think that it is sensible?
>
> We will then mover to other machines and hopefully test fusion and other approaches.
>
> Lorenzo
>
>
> On Mar 1, 2013, at 3:13 PM, David Kelly wrote:
>
> I think it is there. It looks like your memory fixes were committed on 2-2, and the current 0.94 branch is based on a snapshot of trunk from 2-15.
>
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Michael Wilde" <wilde at mcs.anl.gov>, "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, March 1, 2013 12:36:03 PM
> Subject: Re: [Swift-devel] 0.94 release note draft
>
> So I don't think I committed the memory leak fixes to 0.94 and I think
> they should be there.
>
> Mihael
>
> On Fri, 2013-03-01 at 07:06 -0600, David Kelly wrote:
> > Here is the list, based mostly on what I could find in the svn logs.
> >
> >
> >
> > - To behavior of iterate has changed from 0.93 to 0.94. If you have scripts
> > that use iterate, please read
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_iterate
> >
> >
> > - Walltimes are more strictly enforced by coasters. Previous to Swift version
> > 0.94, if an application run with coasters would exceed its specified
> > maxwalltime, it would be allowed to continue to execute. However, if this
> > would cause the worker on which the application was running to exceed its
> > maxwalltime, the queuing system would kill the worker. The resulting error
> > message was not always very clear. Since version 0.94 coaster workers enforce
> > the user-specified maxwalltime. If an application exceeds its maxwalltime,
> > the coaster worker will not allow it to continue, but terminate it and report
> > the error.
> >
> >
> > - Swift will now use camel case for functions, for example, @toInt instead of
> > @toint. The previous naming convention will still work, but you may see
> > deprecated warnings.
> >
> >
> > - Associative arrays have been added. More details and examples can be found at
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_associative_arrays
> >
> >
> > - Dynamic profiles. Many settings formerly only definable in sites.xml can now
> > be set on a per-app basis. This can make things easier when running
> > multiple apps that have different requirements for settings like processors
> > per node and wallitme.
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_dynamic_profiles
> >
> >
> > - Added a new ssh command line provider. Previously ssh support was done by
> > creating a file called ~/.ssh/auth.defaults. The ssh command line provider
> > is more flexible and doesn't require this step. ssh-cl allows you to use
> > SSH agents. You can use ssh-cl by adding something like this to your sites.xml:
> >
> >
> > <execution provider="coaster" url="my.host.uchicago.edu" jobmanager="ssh-cl:pbs"/>
> >
> >
> > - Many fixes and improvements to improve the reliability and performance of
> > coaster provider staging.
> >
> >
> > - Added support for the Slurm scheduler
> >
> >
> > - Added support for the LSF scheduler
> >
> >
> > - Improvements to condor provider (non-shared jobtype and more flexibility
> > to define what gets added to the submit script).
> >
> >
> > - Fixes for the textual user interface (TUI). Adding the -tui option to the swift
> > command line allows you to monitor progress in a curses based menu. A brief
> > example of this can be found at http://www.ci.uchicago.edu/~davidk/modis.ogv.
> >
> >
> > - Added the ability to call Java methods within swift using @java. For example:
> > float f = @java("java.lang.Math", "sin", 0.5);
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_java
> >
> >
> > - Added a hang checker that provides the user with more information about
> > potential hangs
> >
> >
> > - @strjoin function for joining strings.
> > http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html#_strjoin
> >
> >
> > - If you have a requirement that a command get run on the worker node before
> > processing any work, worker.pl will now execute commands stored in the
> > environment variable $WORKER_INIT_CMD
> >
> >
> > - Use $SWIFT_USERHOME to determine where to swift should create some of its
> > required files. This defaults to $HOME, but this may cause problems in some
> > situations where $HOME is not accessable on worker nodes.
> >
> >
> > - Experimental "wrapper staging" feature that delegates file staging to an
> > external wrapper script.
> >
> >
> > - Various improvements to the way that Swift runs MPI jobs.
> >
> >
> > - Better OSG integration/support using GlideinWMS.
> >
> >
> > ----- Original Message -----
> >
> >
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Thursday, February 14, 2013 1:08:56 PM
> > Subject: Fwd: 0.94 release note draft
> >
> >
> > Some notes toward an 0.94 release notes document.
> >
> > There's a longer list, I think in an IM chat transcript, that we need to incorporate.
> >
> > Please send additional items to this thread for David to integrate.
> >
> > Thanks,
> >
> > - Mike
>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130301/6909f511/attachment.html>
More information about the Swift-devel
mailing list