[Swift-user] How does swift mapping to files that do not yet exist work?

[Michael Wilde]

Jonathan,

I tried running your script under a fairly old version of trunk, and it seems to work fine for me with the params you specified below. I used a simple shell script of echoes instead of a matlab script.

Swift should know that the file variable "output" is an output variable, and not expect it to exist before the app is run. And indeed, in my test of your script this works fine.

What version of Swift are you running? Can you post your swift log file, and the full swift output from stdout/err?

Is there some delta between what you posted and what you're actually running, perhaps?

I *did* get file-not-found errors for *input* files that were missing, until I provided all the expected input files, but the output file did not pre-exist and was created as expected:

com$ ls -l *.mat
-rw-r--r-- 1 wilde ci-users 12 Jul  3 18:36 SCE_Par.mat
-rw-r--r-- 1 wilde ci-users 47 Jul  3 18:41 new_point1_1.mat
-rw-r--r-- 1 wilde ci-users 47 Jul  3 18:41 new_point2_1.mat
-rw-r--r-- 1 wilde ci-users 20 Jul  3 18:38 simplex1_1.mat
-rw-r--r-- 1 wilde ci-users 20 Jul  3 18:41 simplex2_1.mat
com$ more new*.mat
::::::::::::::
new_point1_1.mat
::::::::::::::
this is the output file named new_point1_1.mat
::::::::::::::
new_point2_1.mat
::::::::::::::
this is the output file named new_point2_1.mat
com$ 


- Mike




----- Original Message -----
> From: "Jonathan Margoliash" <jmargolpeople at gmail.com>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: swift-user at ci.uchicago.edu
> Sent: Tuesday, July 3, 2012 5:24:19 PM
> Subject: Re: [Swift-user] How does swift mapping to files that do not yet exist work?
> Sorry, I should have specified this earlier, but I do not believe my
> cce_ua function was ever called. The first line of my cce_ua function
> is a statement that prints to stdout. The second and third lines open
> a file (via absolute path) and write to it. I have not observed either
> of those being executed. Furthermore, the following (matlab) code
> process the output argument:
> 
> output_point_location = argv(){3};
> 
> new_point = ....
> 
> save(output_point_location, 'new_point');
> 
> There is no other code that modifies the output_point_location
> variable. So shouldn't this work, given how swift passes files as
> command line arguments?
> 
> I think something else is the problem, but of course any solution
> would be great. Thanks!
> 
> Jonathan
> 
> 
> On Tue, Jul 3, 2012 at 5:11 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> Jonathan, from a cursory read of your script, I suspect that the Swift
> runtime is complaining that the app cce_ua did not produce the file
> new_point1_1.mat, which your output file mapping specified.
> 
> Given your output mapping, Swift is expecting new_point1_1.mat to
> exist in the temporary job directory in which your app was launched. I
> suspect your app (matlab script) did not create a file of this exact
> name.
> 
> You can specify the property "sitedir.keep=true" in your config file
> (e.g., -config cf on the command line) to tell Swift to retain these
> job directories on failure. They should be below the "workdirectory"
> that you specified in your sites.xml file.
> (I think...)
> 
> - Mike
> 
> 
> 
> 
> ----- Original Message -----
> > From: "Jonathan Margoliash" < jmargolpeople at gmail.com >
> > To: swift-user at ci.uchicago.edu
> > Sent: Tuesday, July 3, 2012 4:48:16 PM
> > Subject: [Swift-user] How does swift mapping to files that do not
> > yet exist work?
> > Hello swift user base,
> >
> > I have been having trouble with a bit of swift code for a while now,
> > and despite my hesitancy to ask other people to debug my code, I
> > believe my issue may be more of a lack of understanding of swift
> > than
> > a simple syntax error, so I have decided to send out this email.
> > Specifically, I have a single_file_mapper mapping to a file which
> > does
> > not yet exist when my code is run, in hopes that the file being
> > mapped
> > to will be written to by a an app call. However, when I try running
> > my
> > code, I get the error back from swift
> >
> > Execution failed:
> > File not found: filename (in this case new_point1_1.mat)
> >
> > This paradigm of referencing files that do not yet exist has worked
> > before in swift, so I do not know what the problem is here.
> > Hopefully
> > it is not a simple syntax bug. Anyway, thank you for your help,
> >
> > Jonathan
> >
> > ----- My Code: ----------
> >
> > #cce_ua.swift
> > type file;
> >
> > int num_complexes = @toint(@arg("num_complexes"));
> > int num_simplexes_per_complex =
> > @toint(@arg("num_simplexes_per_complex"));
> >
> > app (file output, file log) cce_ua (file input1, file input2, int
> > number, file input3)
> > {
> > cce_ua @input1 @input2 @output number @input3 stdout=@filename(log)
> > stderr=@filename(log);
> > #located at _RUNDIR_/swat/cce_ua.sh
> > }
> >
> > file inSCE<single_file_mapper;file="SCE_Par.mat">;
> > file inrunswat<single_file_mapper;file="RunSwat.m">;
> >
> > foreach i in [1:num_complexes]
> > {
> > foreach j in [1:num_simplexes_per_complex]
> > {
> > # These files already exist in the local directory
> > file input<single_file_mapper; file=@strcat("simplex", i, "_", j,
> > ".mat")>;
> > # These files do not exist in the local directory at runtime
> > file output<single_file_mapper; file=@strcat("new_point", i, "_", j,
> > ".mat")>;
> > file log<single_file_mapper; file=@strcat("cce_ua", i, "_", j,
> > "_log.txt")>;
> >
> > (output, log) = cce_ua(inSCE, input, i*num_simplexes_per_complex +
> > j,
> > inrunswat);
> > }
> > }
> >
> >
> > ---- The command line call to swift (broken up over multiple lines)
> > ---
> > swift -tc.file swift_files/executable_locations -sites.file
> > swift_files/sites.xml cce_ua.swift -num_complexes=2
> > -num_simplexes_per_complex=1
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory