[Swift-user] Resuming jobs when script has changed
Mihael Hategan
hategan at mcs.anl.gov
Tue May 20 01:41:03 CDT 2014
Hi Greg,
So I think I might be misunderstanding the issue. And I am not talking
about what you have right now, but what you want to get to. So let me
make some toy example and see if it's anything like what you need.
There is some directory. Let's call it "d". In "d", you run swift. It
produces, among other things, some files, let's call them "b??". The
basic mechanism for producing the "b??" files is something of this sort
(in swift):
file[] b <someMapperThatProducesFilesOfTheForm_b??>;
foreach i in INDEX_LIST {
b[i] = generate(i);
}
Let's say that when you first run swift, you get b00, b05, and b08.
These files are subsequently used to do more complex stuff, say "c =
f(b[i])".
Now, in a subsequent run, INDEX_LIST is {02, 05, 07}. You would like b05
to not be generated again, but the old one to be used.
Is this correct?
I'm getting a bit confused by your "swift intermediate directory" and
how it relates to this picture.
Mihael
On Tue, 2014-05-20 at 06:18 +0000, Bronevetsky, Greg wrote:
> I'm not sure I'm following. In my case mydir/ and olddir/ are the
> same. I'm trying to run the script in its original directory within my
> home directory, while making sure not to re-create files that already
> exist. The issue that I had is that in my original solution I copied
> files from my home directory to Swift's intermediate directory and
> then back, overwriting the original file in my home directory with an
> identical copy. The first copy (home -> Swift intermediate dir) seems
> unavoidable but the second seems unnecessary.
>
> Greg Bronevetsky
> Lawrence Livermore National Lab
> (925) 424-5756
> bronevetsky at llnl.gov
> http://greg.bronevetsky.com
>
>
> -----Original Message-----
> From: Mihael Hategan [mailto:hategan at mcs.anl.gov]
> Sent: Monday, May 19, 2014 11:03 PM
> To: Bronevetsky, Greg
> Cc: swift-user at ci.uchicago.edu
> Subject: Re: [Swift-user] Resuming jobs when script has changed
>
> Sorry. I thought you didn't want the overhead of a copy.
>
> If you do want them in your directory, then map "copy" explicitly to the relevant name.
>
> In other words:
>
> file a <"mydir/a">;
> file olda <"olddir/a">;
> a = olda;
>
> will copy olddir/a to mydir/a
>
> but
>
> file a;
> file olda <"olddir/a">;
> a = olda;
>
> will not.
>
> Mihael
>
>
> On Tue, 2014-05-20 at 05:30 +0000, Bronevetsky, Greg wrote:
> > Thanks Mihael. I just tried using your operations in my test code (below). It seems to detect whether the files do/don't exist and seems to try to generate the files but they never appear in my directory. During a fresh run in an empty directory I get the following output:
> >
> > Swift 0.94.1 swift-r7114 cog-r3803
> >
> > RunID: 20140519-2230-4lyyh8me
> > Progress: time: Mon, 19 May 2014 22:30:03 -0700
> > Progress: time: Mon, 19 May 2014 22:30:04 -0700 Stage in:1
> > Submitting:1
> > Progress: time: Mon, 19 May 2014 22:30:09 -0700 Active:1 Checking
> > status:1
> > path=g/g15/bronevet/apps/swift-0.94.1/examples/test/work/dataF,
> > exists=false Generating
> > file://localhost/_concurrent/data-5fbfabc7-8f5d-4240-82b1-4c7ecfdc33aa
> > -path=g/g15/bronevet/apps/swift-0.94.1/examples/test/work/copyF,
> > exists=false Generating
> > file://localhost/_concurrent/copy-44d394f6-53f3-4f5b-9ec9-fa2c19f5fcdf
> > - from
> > file://localhost/_concurrent/data-5fbfabc7-8f5d-4240-82b1-4c7ecfdc33aa
> > -Final status: Mon, 19 May 2014 22:30:09 -0700 Finished
> > successfully:3
> >
> > Greg Bronevetsky
> > Lawrence Livermore National Lab
> > (925) 424-5756
> > bronevetsky at llnl.gov
> > http://greg.bronevetsky.com
> >
> > import "common";
> > type file;
> >
> > string
> > WORK_PATH="/g/g15/bronevet/apps/swift-0.94.1/examples/test/work";
> >
> > app (file outF) writeFile(string message) {
> > echo message stdout=@filename(outF); }
> >
> > app (file outF) copyFile(file inF) {
> > cp @filename(inF) @filename(outF);
> > }
> >
> > file data;
> > file dataOld <single_file_mapper; file="dataF">;
> > if(!exists(@strcat(WORK_PATH, "/dataF"))) {
> > tracef("Generating %M\n", data);
> > (data) = writeData("hello");
> > } else
> > { data = dataOld; }
> >
> > file copy;
> > file copyOld <single_file_mapper; file="copyF">;
> > if(!exists(@strcat(WORK_PATH, "/copyF"))) {
> > tracef("Generating %M from %M\n", copy, data);
> > (copy) = copyFile(data);
> > } else
> > { copy = copyOld; }
> >
> >
> > -----Original Message-----
> > From: Mihael Hategan [mailto:hategan at mcs.anl.gov]
> > Sent: Monday, May 19, 2014 4:25 PM
> > To: Bronevetsky, Greg
> > Cc: swift-user at ci.uchicago.edu
> > Subject: Re: [Swift-user] Resuming jobs when script has changed
> >
> > Here's the swift part. It will only work with absolute paths. It can probably be modified to only work with relative paths.
> >
> > type file;
> >
> > app (file outf) existsApp(string path) {
> > existsApp path stdout=@filename(outf); }
> >
> > (boolean result) exists(string path) {
> > result = readData(existsApp(path));
> > }
> >
> > file there <"/bin/bash">;
> > file notthere <"/bin/notthere">;
> >
> > tracef("Y %b\n", exists(@filename(there))); tracef("N %b\n",
> > exists(@filename(notthere)));
> >
> > And existsApp:
> > #!/bin/bash
> >
> > # add leading / since @filename removes it # this will not work with swift > 0.94 if [ -f "/$1" ]; then
> > echo "true"
> > else
> > echo "false"
> > fi
> >
> >
> >
> > On Mon, 2014-05-19 at 15:28 -0700, Mihael Hategan wrote:
> > > On Mon, 2014-05-19 at 22:21 +0000, Bronevetsky, Greg wrote:
> > > > Thanks, Mihael! I've modified my original code based on your suggestions (new code below) but I'm getting the following error:
> > > > Could not start execution
> > > > Procedure exists is not defined.
> > >
> > > Sorry. That's where your implementation of exists for absolute path
> > > names goes. I'll send you the relevant code shortly. I'm working on
> > > the array splitting issue now.
> > >
> > > Mihael
> > >
> > >
> > > _______________________________________________
> > > Swift-user mailing list
> > > Swift-user at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> >
>
>
More information about the Swift-user
mailing list