[Swift-user] assign temporary file to mapper

Thu Nov 29 23:02:38 CST 2012

Hao, my apologies - I lost track of your problem.

Can you send your log file from this run? it should be a file with your scriptname, then -20121120-1538-mt4hjjd6.log.

Something is inconsistent here: your output shows trace lines that are not in the source code you sent.  The log file will have the source code that matches the run which produced there error.

I may be missing something here, but it looks like you are concatenating two mapped temporary filenames and mapping yet another filename to that value:

File not found: /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--\
array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-

But I dont see where this is happening the source code you sent.

- Mike

----- Original Message -----
> From: "Hao Yang" <yanghao0614 at gmail.com>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: mosastore at googlegroups.com, swift-user at ci.uchicago.edu
> Sent: Tuesday, November 20, 2012 10:33:15 PM
> Subject: Re: [Swift-user] assign temporary file to mapper
> Hi, Michael:
> 
> 
> The output_chunks[2] is assigned in the iterate via output_chunks[i] =
> appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the
> script code and the error message below).
> 
> 
> // swift code
> type RSFile; // rsf file
> type StringFile; // File that stores strings
> 
> 
> // APPs
> // getNDims: Extract last dimension from rsf file
> app (StringFile ndims) appGetNDims(RSFile input) {
> sffiledims_script @filename(input) "1" stdout=@filename(ndims);
> }
> 
> 
> // getDims: Extract dimension list from rsf file
> app (StringFile dims) appGetDims(RSFile input) {
> sffiledims_script @filename(input) "2" stdout=@filename(dims);
> }
> 
> 
> // getWindow: Window a slice from rsf file
> app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) {
> sfwindow_script @filename(inp) @filename(out) dim slice;
> }
> 
> 
> // addFiles: adds inp1 and inp2 and returns out
> app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) {
> sfadd_script @filename(out) @filename(inp1) @filename(inp2);
> }
> 
> 
> // In-core variables
> string input_name; // input name of the file
> string output_name; // output name of the file
> string last_dim; // Last dimension of file (default splitting
> dimension)
> string split_dim; // Splitting dimension
> string dims_list; // string of dimensions (unsplitted)
> string dims_array[]; // Array of string of dimensions (splitted)
> int dim_length; // length of splitted dimension
> 
> 
> // File variables
> // Setup input and output files
> RSFile input_data <single_file_mapper; file=input_name>;
> RSFile output_data <single_file_mapper; file=output_name>;
> StringFile ndims_file; // number of dimensions of the input file
> StringFile dims_file; // dimension array of the input file
> RSFile input_chunks[]; // Splitted input
> RSFile processed_chunks[]; // data chunks after parallel processing
> RSFile output_chunks[]; // Temporary output array for reduction
> 
> 
> // main part
> // Define and parse input arguments
> input_name = @arg("in","in.rsf");
> output_name = @arg("out","out.rsf");
> 
> 
> // extract dims and setup variables
> ndims_file = appGetNDims(input_data);
> dims_file = appGetDims(input_data);
> 
> 
> // Define and parse split dimension and dimensions array
> last_dim = readData(ndims_file);
> split_dim = @arg("dim", last_dim);
> dims_list = readData(dims_file);
> dims_array = @strsplit(dims_list,",");
> dim_length = @toint(dims_array[@toint(split_dim)-1]);
> trace("Dim length = ", dim_length);
> 
> 
> // Split the files parallelly
> foreach i in [0:dim_length-1] {
> trace("Splitting ", i);
> input_chunks[i] = appSplitFile(input_data, split_dim, i+1);
> trace("Working furiously on ", @filename(input_chunks[i]));
> trace("Chunk number: ", i);
> processed_chunks[i] = input_chunks[i];
> }
> 
> 
> // Add all the chunks together
> iterate i {
> if( i == 0 ) {
> trace("Setting chunk 0 to ", @filename(output_chunks[0]));
> output_chunks[0] = processed_chunks[0];
> } else {
> trace("Adding ", i);
> output_chunks[i] = appAddFiles(processed_chunks[i],
> output_chunks[i-1]);
> }
> } until (i == dim_length-1);
> 
> 
> output_data = output_chunks[dim_length-1];
> 
> 
> // swift message
> 
> RunID: 20121120-1538-mt4hjjd6
> Progress: time: Tue, 20 Nov 2012 15:38:03 -0800
> SwiftScript trace: Setting chunk 0 to ,
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0
> SwiftScript trace: filename of output_chunks[0] is,
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0
> SwiftScript trace: filename of processed_chunks[0] is,
> _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0
> Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1 Finished
> successfully:1
> SwiftScript trace: Dim length = , 3.0
> SwiftScript trace: Splitting , 1.0
> 
> SwiftScript trace: Chunk number: , 1.0
> SwiftScript trace: Chunk number: , 2.0
> SwiftScript trace: Working furiously on ,
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1
> 
> SwiftScript trace: Splitting , 0.0
> SwiftScript trace: Chunk number: , 0.0
> SwiftScript trace: Splitting , 2.0
> SwiftScript trace: Working furiously on ,
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2
> SwiftScript trace: Working furiously on ,
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0
> Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1 Stage
> in:1 Finished successfully:3
> Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1
> Finished successfully:4
> SwiftScript trace: Adding , 1.0
> SwiftScript trace: filename of output_chunks[i] is,
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1
> SwiftScript trace: filename of output_chunks[i-1] is,
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0
> SwiftScript trace: filename of processed_chunks[i] is,
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1
> SwiftScript trace: Adding , 2.0
> SwiftScript trace: filename of processed_chunks[i] is,
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2
> SwiftScript trace: filename of output_chunks[i] is,
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2
> SwiftScript trace: filename of output_chunks[i-1] is,
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1
> Execution failed:
> File not found:
> /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2
> 
> 
> From the trace we can see filename of output_chunks[2] is
> _concurrent/output_chunks- 33fce3ec-f16c-4430-b086-
> cc90f611a0a2--array//elt-2 (we also verified the file is there after
> the execution failure).
> 
> 
> Thank you.
> 
> On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> Hao,
> 
> Are you sure you assigned a value to output_chunks[2]?
> 
> Can you post the entire program, and the entire error message?
> 
> Thanks,
> 
> - Mike
> 
> 
> 
> ----- Original Message -----
> > From: "Hao Yang" < yanghao0614 at gmail.com >
> > To: swift-user at ci.uchicago.edu
> > Cc: mosastore at googlegroups.com
> > Sent: Tuesday, November 20, 2012 7:22:45 PM
> > Subject: [Swift-user] assign temporary file to mapper
> > Hi, all:
> >
> >
> > I have some problem assigning temporary file to some mapper variable
> > (explained in the code below).
> >
> >
> > type RSFile;
> > string output_name;
> > output_name = @arg("out","out.rsf");
> > RSFile output_data <single_file_mapper; file=output_name>;
> > RSFile output_chunks[];
> > // some calculation to compute output_chunks[0], output_chunks[1],
> > output_chunks[2]
> > // at the end
> > output_data = output_chunks[2];
> >
> >
> >
> > When I try this script, swift reports " File output_chunks[2] not
> > found " error and failed. But the output_chunks[2] is still
> > available
> > in the directory generated by swift ( _ concurrent/output_chunks-
> > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error
> 
> > message be expected?
> >
> >
> > Thank you.
> >
> > --
> > Best Regards,
> > Hao Yang
> >
> > The University of British Columbia
> >
> >
> >
> >
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> 
> 
> 
> 
> --
> Best Regards,
> Hao Yang
> Networked Systems Laboratory
> Electrical and Computer Engineering
> The University of British Columbia

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory