[Swift-user] Resuming jobs when script has changed

[Bronevetsky, Greg]

I'm not sure I'm following. In my case mydir/ and olddir/ are the same. I'm trying to run the script in its original directory within my home directory, while making sure not to re-create files that already exist. The issue that I had is that in my original solution I copied files from my home directory to Swift's intermediate directory and then back, overwriting the original file in my home directory with an identical copy. The first copy (home -> Swift intermediate dir) seems unavoidable but the second seems unnecessary.

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov
http://greg.bronevetsky.com


-----Original Message-----
From: Mihael Hategan [mailto:hategan at mcs.anl.gov] 
Sent: Monday, May 19, 2014 11:03 PM
To: Bronevetsky, Greg
Cc: swift-user at ci.uchicago.edu
Subject: Re: [Swift-user] Resuming jobs when script has changed

Sorry. I thought you didn't want the overhead of a copy.

If you do want them in your directory, then map "copy" explicitly to the relevant name.

In other words:

file a <"mydir/a">;
file olda <"olddir/a">;
a = olda;

will copy olddir/a to mydir/a

but

file a;
file olda <"olddir/a">;
a = olda;

will not.

Mihael


On Tue, 2014-05-20 at 05:30 +0000, Bronevetsky, Greg wrote:
> Thanks Mihael. I just tried using your operations in my test code (below). It seems to detect whether the files do/don't exist and seems to try to generate the files but they never appear in my directory. During a fresh run in an empty directory I get the following output:
> 
> Swift 0.94.1 swift-r7114 cog-r3803
> 
> RunID: 20140519-2230-4lyyh8me
> Progress:  time: Mon, 19 May 2014 22:30:03 -0700
> Progress:  time: Mon, 19 May 2014 22:30:04 -0700  Stage in:1  
> Submitting:1
> Progress:  time: Mon, 19 May 2014 22:30:09 -0700  Active:1  Checking 
> status:1 
> path=g/g15/bronevet/apps/swift-0.94.1/examples/test/work/dataF, 
> exists=false Generating 
> file://localhost/_concurrent/data-5fbfabc7-8f5d-4240-82b1-4c7ecfdc33aa
> -path=g/g15/bronevet/apps/swift-0.94.1/examples/test/work/copyF, 
> exists=false Generating 
> file://localhost/_concurrent/copy-44d394f6-53f3-4f5b-9ec9-fa2c19f5fcdf
> - from 
> file://localhost/_concurrent/data-5fbfabc7-8f5d-4240-82b1-4c7ecfdc33aa
> -Final status: Mon, 19 May 2014 22:30:09 -0700  Finished 
> successfully:3
> 
> Greg Bronevetsky
> Lawrence Livermore National Lab
> (925) 424-5756
> bronevetsky at llnl.gov
> http://greg.bronevetsky.com
> 
> import "common";
> type file;
> 
> string 
> WORK_PATH="/g/g15/bronevet/apps/swift-0.94.1/examples/test/work";
> 
> app (file outF) writeFile(string message) {
>   echo message stdout=@filename(outF); }
> 
> app (file outF) copyFile(file inF) {
>   cp @filename(inF) @filename(outF);
> }
> 
> file data;
> file dataOld <single_file_mapper; file="dataF">; 
> if(!exists(@strcat(WORK_PATH, "/dataF"))) {
>   tracef("Generating %M\n", data);
>   (data) = writeData("hello");
> } else
> { data = dataOld; }
> 
> file copy;
> file copyOld <single_file_mapper; file="copyF">; 
> if(!exists(@strcat(WORK_PATH, "/copyF"))) {
>   tracef("Generating %M from %M\n", copy, data);
>   (copy) = copyFile(data);
> } else
> { copy = copyOld; }
> 
> 
> -----Original Message-----
> From: Mihael Hategan [mailto:hategan at mcs.anl.gov]
> Sent: Monday, May 19, 2014 4:25 PM
> To: Bronevetsky, Greg
> Cc: swift-user at ci.uchicago.edu
> Subject: Re: [Swift-user] Resuming jobs when script has changed
> 
> Here's the swift part. It will only work with absolute paths. It can probably be modified to only work with relative paths.
> 
> type file;
> 
> app (file outf) existsApp(string path) {
>   existsApp path stdout=@filename(outf); }
> 
> (boolean result) exists(string path) {
>   result = readData(existsApp(path));
> }
> 
> file there <"/bin/bash">;
> file notthere <"/bin/notthere">;
> 
> tracef("Y %b\n", exists(@filename(there))); tracef("N %b\n", 
> exists(@filename(notthere)));
> 
> And existsApp:
> #!/bin/bash
> 
> # add leading / since @filename removes it # this will not work with swift > 0.94 if [ -f "/$1" ]; then
>         echo "true"
> else
>         echo "false"
> fi
> 
> 
> 
> On Mon, 2014-05-19 at 15:28 -0700, Mihael Hategan wrote:
> > On Mon, 2014-05-19 at 22:21 +0000, Bronevetsky, Greg wrote:
> > > Thanks, Mihael! I've modified my original code based on your suggestions (new code below) but I'm getting the following error:
> > > Could not start execution
> > >         Procedure exists is not defined.
> > 
> > Sorry. That's where your implementation of exists for absolute path 
> > names goes. I'll send you the relevant code shortly. I'm working on 
> > the array splitting issue now.
> > 
> > Mihael
> > 
> > 
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
>