[Swift-user] assign temporary file to mapper

Fri Nov 30 17:21:22 CST 2012

Dear Hao,

Mihael has diagnosed this problem as likely being caused by a Swift bug in 0.93 that was fixed many months ago in trunk.  Can you repeat your test with the current 0.94 release candidate? It at:

 http://www.ci.uchicago.edu/swift/packages/swift-0.94RC1.tar.gz

Thanks, and sorry again for the delay in debugging this.

- Mike

----- Original Message -----
> From: "Hao Yang" <yanghao0614 at gmail.com>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: mosastore at googlegroups.com, swift-user at ci.uchicago.edu
> Sent: Friday, November 30, 2012 1:32:23 AM
> Subject: Re: [Swift-user] assign temporary file to mapper
> Hi, Michael:
> 
> 
> Please find the log file in the attachment.
> 
> In the source code we declared a temporary file array RSFile
> output_chunks[] , and a single file mapper. RSFile output_data
> <single_file_mapper; file=output_name> . Then we used the iterate at
> the end of the source code to get output_chunks[0], ..., till
> output_chunks[dim_length - 1]. The error happened when Swift tried to
> execute the last line output_data = output_chunks[dim_length-1]. It
> couldn't find the file output_chunks[dim_length-1], but we verified
> the file was available in the _concurrent directory. So we were
> wondering if it is valid to assign temporary file to file mapper.
> 
> 
> Thank you.
> 
> 
> On Thu, Nov 29, 2012 at 9:02 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> Hao, my apologies - I lost track of your problem.
> 
> Can you send your log file from this run? it should be a file with
> your scriptname, then -20121120-1538-mt4hjjd6.log.
> 
> Something is inconsistent here: your output shows trace lines that are
> not in the source code you sent. The log file will have the source
> code that matches the run which produced there error.
> 
> I may be missing something here, but it looks like you are
> concatenating two mapped temporary filenames and mapping yet another
> filename to that value:
> 
> File not found:
> /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--\
> array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-
> 
> But I dont see where this is happening the source code you sent.
> 
> 
> - Mike
> 
> ----- Original Message -----
> > From: "Hao Yang" < yanghao0614 at gmail.com >
> 
> 
> > To: "Michael Wilde" < wilde at mcs.anl.gov >
> > Cc: mosastore at googlegroups.com , swift-user at ci.uchicago.edu
> > Sent: Tuesday, November 20, 2012 10:33:15 PM
> > Subject: Re: [Swift-user] assign temporary file to mapper
> > Hi, Michael:
> >
> >
> > The output_chunks[2] is assigned in the iterate via output_chunks[i]
> > =
> > appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the
> > script code and the error message below).
> >
> >
> > // swift code
> > type RSFile; // rsf file
> > type StringFile; // File that stores strings
> >
> >
> > // APPs
> > // getNDims: Extract last dimension from rsf file
> > app (StringFile ndims) appGetNDims(RSFile input) {
> > sffiledims_script @filename(input) "1" stdout=@filename(ndims);
> > }
> >
> >
> > // getDims: Extract dimension list from rsf file
> > app (StringFile dims) appGetDims(RSFile input) {
> > sffiledims_script @filename(input) "2" stdout=@filename(dims);
> > }
> >
> >
> > // getWindow: Window a slice from rsf file
> > app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) {
> > sfwindow_script @filename(inp) @filename(out) dim slice;
> > }
> >
> >
> > // addFiles: adds inp1 and inp2 and returns out
> > app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) {
> > sfadd_script @filename(out) @filename(inp1) @filename(inp2);
> > }
> >
> >
> > // In-core variables
> > string input_name; // input name of the file
> > string output_name; // output name of the file
> > string last_dim; // Last dimension of file (default splitting
> > dimension)
> > string split_dim; // Splitting dimension
> > string dims_list; // string of dimensions (unsplitted)
> > string dims_array[]; // Array of string of dimensions (splitted)
> > int dim_length; // length of splitted dimension
> >
> >
> > // File variables
> > // Setup input and output files
> > RSFile input_data <single_file_mapper; file=input_name>;
> > RSFile output_data <single_file_mapper; file=output_name>;
> > StringFile ndims_file; // number of dimensions of the input file
> > StringFile dims_file; // dimension array of the input file
> > RSFile input_chunks[]; // Splitted input
> > RSFile processed_chunks[]; // data chunks after parallel processing
> > RSFile output_chunks[]; // Temporary output array for reduction
> >
> >
> > // main part
> > // Define and parse input arguments
> > input_name = @arg("in","in.rsf");
> > output_name = @arg("out","out.rsf");
> >
> >
> > // extract dims and setup variables
> > ndims_file = appGetNDims(input_data);
> > dims_file = appGetDims(input_data);
> >
> >
> > // Define and parse split dimension and dimensions array
> > last_dim = readData(ndims_file);
> > split_dim = @arg("dim", last_dim);
> > dims_list = readData(dims_file);
> > dims_array = @strsplit(dims_list,",");
> > dim_length = @toint(dims_array[@toint(split_dim)-1]);
> > trace("Dim length = ", dim_length);
> >
> >
> > // Split the files parallelly
> > foreach i in [0:dim_length-1] {
> > trace("Splitting ", i);
> > input_chunks[i] = appSplitFile(input_data, split_dim, i+1);
> > trace("Working furiously on ", @filename(input_chunks[i]));
> > trace("Chunk number: ", i);
> > processed_chunks[i] = input_chunks[i];
> > }
> >
> >
> > // Add all the chunks together
> > iterate i {
> > if( i == 0 ) {
> > trace("Setting chunk 0 to ", @filename(output_chunks[0]));
> > output_chunks[0] = processed_chunks[0];
> > } else {
> > trace("Adding ", i);
> > output_chunks[i] = appAddFiles(processed_chunks[i],
> > output_chunks[i-1]);
> > }
> > } until (i == dim_length-1);
> >
> >
> > output_data = output_chunks[dim_length-1];
> >
> >
> > // swift message
> >
> > RunID: 20121120-1538-mt4hjjd6
> > Progress: time: Tue, 20 Nov 2012 15:38:03 -0800
> > SwiftScript trace: Setting chunk 0 to ,
> > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0
> > SwiftScript trace: filename of output_chunks[0] is,
> > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0
> > SwiftScript trace: filename of processed_chunks[0] is,
> > _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0
> > Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1
> > Finished
> > successfully:1
> > SwiftScript trace: Dim length = , 3.0
> > SwiftScript trace: Splitting , 1.0
> >
> > SwiftScript trace: Chunk number: , 1.0
> > SwiftScript trace: Chunk number: , 2.0
> > SwiftScript trace: Working furiously on ,
> > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1
> >
> > SwiftScript trace: Splitting , 0.0
> > SwiftScript trace: Chunk number: , 0.0
> > SwiftScript trace: Splitting , 2.0
> > SwiftScript trace: Working furiously on ,
> > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2
> > SwiftScript trace: Working furiously on ,
> > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0
> > Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1
> > Stage
> > in:1 Finished successfully:3
> > Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1
> > Finished successfully:4
> > SwiftScript trace: Adding , 1.0
> > SwiftScript trace: filename of output_chunks[i] is,
> > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1
> > SwiftScript trace: filename of output_chunks[i-1] is,
> > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0
> > SwiftScript trace: filename of processed_chunks[i] is,
> > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1
> > SwiftScript trace: Adding , 2.0
> > SwiftScript trace: filename of processed_chunks[i] is,
> > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2
> > SwiftScript trace: filename of output_chunks[i] is,
> > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2
> > SwiftScript trace: filename of output_chunks[i-1] is,
> > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1
> > Execution failed:
> > File not found:
> > /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2
> >
> >
> > From the trace we can see filename of output_chunks[2] is
> > _concurrent/output_chunks- 33fce3ec-f16c-4430-b086-
> > cc90f611a0a2--array//elt-2 (we also verified the file is there after
> > the execution failure).
> >
> >
> > Thank you.
> >
> > On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde < wilde at mcs.anl.gov >
> > wrote:
> >
> >
> > Hao,
> >
> > Are you sure you assigned a value to output_chunks[2]?
> >
> > Can you post the entire program, and the entire error message?
> >
> > Thanks,
> >
> > - Mike
> >
> >
> >
> > ----- Original Message -----
> > > From: "Hao Yang" < yanghao0614 at gmail.com >
> > > To: swift-user at ci.uchicago.edu
> > > Cc: mosastore at googlegroups.com
> > > Sent: Tuesday, November 20, 2012 7:22:45 PM
> > > Subject: [Swift-user] assign temporary file to mapper
> > > Hi, all:
> > >
> > >
> > > I have some problem assigning temporary file to some mapper
> > > variable
> > > (explained in the code below).
> > >
> > >
> > > type RSFile;
> > > string output_name;
> > > output_name = @arg("out","out.rsf");
> > > RSFile output_data <single_file_mapper; file=output_name>;
> > > RSFile output_chunks[];
> > > // some calculation to compute output_chunks[0], output_chunks[1],
> > > output_chunks[2]
> > > // at the end
> > > output_data = output_chunks[2];
> > >
> > >
> > >
> > > When I try this script, swift reports " File output_chunks[2] not
> > > found " error and failed. But the output_chunks[2] is still
> > > available
> > > in the directory generated by swift ( _ concurrent/output_chunks-
> > > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error
> >
> > > message be expected?
> > >
> > >
> > > Thank you.
> > >
> > > --
> > > Best Regards,
> > > Hao Yang
> > >
> > > The University of British Columbia
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Swift-user mailing list
> > > Swift-user at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >
> >
> >
> >
> >
> > --
> > Best Regards,
> > Hao Yang
> > Networked Systems Laboratory
> > Electrical and Computer Engineering
> > The University of British Columbia
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> 
> 
> 
> 
> --
> Best Regards,
> Hao Yang
> Networked Systems Laboratory
> Electrical and Computer Engineering
> The University of British Columbia

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory