[Swift-user] Resuming jobs when script has changed

[Bronevetsky, Greg]

I'm using Swift 0.94 so I'll be happy to switch to the built-in exists function. However, I don't think resolves the extra copying issue. The reason why I had to go the way I did is that I had to make sure that regardless of whether a file exists or not, the Swift variable that represents it needs to be the output of some task, otherwise Swift can't track dependencies on it. The only way I could think of to achieve this while, making sure that the file associated with the Swift-visible variable had the right contents, was to copy the data from the original to the Swift-visible file. However Swift then copies the file back to the original since I gave them the same names. 

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov
http://greg.bronevetsky.com


-----Original Message-----
From: Mihael Hategan [mailto:hategan at mcs.anl.gov] 
Sent: Monday, May 19, 2014 2:44 PM
To: Bronevetsky, Greg
Cc: swift-user at ci.uchicago.edu
Subject: Re: [Swift-user] Resuming jobs when script has changed

On Mon, 2014-05-19 at 19:06 +0000, Bronevetsky, Greg wrote:

> -          I need to explicitly copy the files that already exist from my working directory to temporary directory used by Swift to enable Swift to recognize these files' existence. Then Swift copies them back to their original locations, which is wasteful but at least correct.
> If anybody can suggest alternatives that overcome the above issues, I'd be grateful. I expect that mine is a common use-case because without this every small change to a script causes all of its intermediate results to be recomputed, even if the script is large and takes days to compute.

If you are using swift > 0.94, you could use the built-in exists() function I mentioned in my previous email.

If not, you can make sure that the fileExists app only runs locally (by only having it for localhost in the sites file). Then, you can pass the working directory as a string to it and modify it to use that directory and the file whose existence you are trying to determine to build an absolute path. That would not require any copying.

Mihael