[Swift-user] Resuming jobs when script has changed

[Bronevetsky, Greg]

The external mapper mechanism looks to be appropriate. To make sure I understand you, is the idea that I'd replace my current mappers which are
<single_file_mapper; file=@strcat("a","b","c")>;
with an external mapper
               < ext; exec="script.sh", arg=@strcat("a","b","c")>;
Then script.sh would return the string @strcat("a","b","c") if the file with this path does not exist and an empty file it does.

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com

From: swift-user-bounces at ci.uchicago.edu [mailto:swift-user-bounces at ci.uchicago.edu] On Behalf Of Michael Wilde
Sent: Friday, May 16, 2014 10:53 AM
To: swift-user at ci.uchicago.edu
Subject: Re: [Swift-user] Resuming jobs when script has changed

Hi Greg,

The Swift resume mechanism can't do this, as its driven by internal variables name within a Swift run, not by external file existence.

The best way to do what you describe below is to write an external input file mapper (e.g. a simple shell or py script) that returns only the files that still need to be produced or processed.  (This is the "ext" mapper: http://swift-lang.org/guides/release-0.94/userguide/userguide.html#_external_mapper )

You can also call a local app to determine what files need to be produced, then use readData() to read those file names into an array, and array_mapper to map an array of remaining work to do.

Would one of these approaches meet your needs?

- Mike

On 5/16/14, 12:35 PM, Bronevetsky, Greg wrote:

I have Swift scripts that scan some portion of a design space and after I have scanned a sub-space of all the possibilities I modify the script to target a different, overlapping portion of the design space. However, it seems that when I do this Swift ignores the fact that I've already computed many of the tasks in the new run when I performed the prior run, and re-executes them redundantly. Is there a way for me to avoid such redundant executions? Can the -resume flag be used in this case?



Greg Bronevetsky

Lawrence Livermore National Lab

(925) 424-5756

bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov><mailto:bronevetsky at llnl.gov><mailto:bronevetsky at llnl.gov>

http://greg.bronevetsky.com








_______________________________________________

Swift-user mailing list

Swift-user at ci.uchicago.edu<mailto:Swift-user at ci.uchicago.edu>

https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user



--

Michael Wilde

Mathematics and Computer Science          Computation Institute

Argonne National Laboratory               The University of Chicago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140516/43ce118b/attachment.html>