[Swift-user] assign temporary file to mapper

[Hao Yang]

Hi, Michael:

Thank you and Mihael for your help. We will try 0.94.

On Fri, Nov 30, 2012 at 3:21 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> Dear Hao,
>
> Mihael has diagnosed this problem as likely being caused by a Swift bug in
> 0.93 that was fixed many months ago in trunk.  Can you repeat your test
> with the current 0.94 release candidate? It at:
>
>  http://www.ci.uchicago.edu/swift/packages/swift-0.94RC1.tar.gz
>
> Thanks, and sorry again for the delay in debugging this.
>
> - Mike
>
> ----- Original Message -----
> > From: "Hao Yang" <yanghao0614 at gmail.com>
> > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > Cc: mosastore at googlegroups.com, swift-user at ci.uchicago.edu
> > Sent: Friday, November 30, 2012 1:32:23 AM
> > Subject: Re: [Swift-user] assign temporary file to mapper
> > Hi, Michael:
> >
> >
> > Please find the log file in the attachment.
> >
> > In the source code we declared a temporary file array RSFile
> > output_chunks[] , and a single file mapper. RSFile output_data
> > <single_file_mapper; file=output_name> . Then we used the iterate at
> > the end of the source code to get output_chunks[0], ..., till
> > output_chunks[dim_length - 1]. The error happened when Swift tried to
> > execute the last line output_data = output_chunks[dim_length-1]. It
> > couldn't find the file output_chunks[dim_length-1], but we verified
> > the file was available in the _concurrent directory. So we were
> > wondering if it is valid to assign temporary file to file mapper.
> >
> >
> > Thank you.
> >
> >
> > On Thu, Nov 29, 2012 at 9:02 PM, Michael Wilde < wilde at mcs.anl.gov >
> > wrote:
> >
> >
> > Hao, my apologies - I lost track of your problem.
> >
> > Can you send your log file from this run? it should be a file with
> > your scriptname, then -20121120-1538-mt4hjjd6.log.
> >
> > Something is inconsistent here: your output shows trace lines that are
> > not in the source code you sent. The log file will have the source
> > code that matches the run which produced there error.
> >
> > I may be missing something here, but it looks like you are
> > concatenating two mapped temporary filenames and mapping yet another
> > filename to that value:
> >
> > File not found:
> >
> /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--\
> >
> array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-
> >
> > But I dont see where this is happening the source code you sent.
> >
> >
> > - Mike
> >
> > ----- Original Message -----
> > > From: "Hao Yang" < yanghao0614 at gmail.com >
> >
> >
> > > To: "Michael Wilde" < wilde at mcs.anl.gov >
> > > Cc: mosastore at googlegroups.com , swift-user at ci.uchicago.edu
> > > Sent: Tuesday, November 20, 2012 10:33:15 PM
> > > Subject: Re: [Swift-user] assign temporary file to mapper
> > > Hi, Michael:
> > >
> > >
> > > The output_chunks[2] is assigned in the iterate via output_chunks[i]
> > > =
> > > appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the
> > > script code and the error message below).
> > >
> > >
> > > // swift code
> > > type RSFile; // rsf file
> > > type StringFile; // File that stores strings
> > >
> > >
> > > // APPs
> > > // getNDims: Extract last dimension from rsf file
> > > app (StringFile ndims) appGetNDims(RSFile input) {
> > > sffiledims_script @filename(input) "1" stdout=@filename(ndims);
> > > }
> > >
> > >
> > > // getDims: Extract dimension list from rsf file
> > > app (StringFile dims) appGetDims(RSFile input) {
> > > sffiledims_script @filename(input) "2" stdout=@filename(dims);
> > > }
> > >
> > >
> > > // getWindow: Window a slice from rsf file
> > > app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) {
> > > sfwindow_script @filename(inp) @filename(out) dim slice;
> > > }
> > >
> > >
> > > // addFiles: adds inp1 and inp2 and returns out
> > > app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) {
> > > sfadd_script @filename(out) @filename(inp1) @filename(inp2);
> > > }
> > >
> > >
> > > // In-core variables
> > > string input_name; // input name of the file
> > > string output_name; // output name of the file
> > > string last_dim; // Last dimension of file (default splitting
> > > dimension)
> > > string split_dim; // Splitting dimension
> > > string dims_list; // string of dimensions (unsplitted)
> > > string dims_array[]; // Array of string of dimensions (splitted)
> > > int dim_length; // length of splitted dimension
> > >
> > >
> > > // File variables
> > > // Setup input and output files
> > > RSFile input_data <single_file_mapper; file=input_name>;
> > > RSFile output_data <single_file_mapper; file=output_name>;
> > > StringFile ndims_file; // number of dimensions of the input file
> > > StringFile dims_file; // dimension array of the input file
> > > RSFile input_chunks[]; // Splitted input
> > > RSFile processed_chunks[]; // data chunks after parallel processing
> > > RSFile output_chunks[]; // Temporary output array for reduction
> > >
> > >
> > > // main part
> > > // Define and parse input arguments
> > > input_name = @arg("in","in.rsf");
> > > output_name = @arg("out","out.rsf");
> > >
> > >
> > > // extract dims and setup variables
> > > ndims_file = appGetNDims(input_data);
> > > dims_file = appGetDims(input_data);
> > >
> > >
> > > // Define and parse split dimension and dimensions array
> > > last_dim = readData(ndims_file);
> > > split_dim = @arg("dim", last_dim);
> > > dims_list = readData(dims_file);
> > > dims_array = @strsplit(dims_list,",");
> > > dim_length = @toint(dims_array[@toint(split_dim)-1]);
> > > trace("Dim length = ", dim_length);
> > >
> > >
> > > // Split the files parallelly
> > > foreach i in [0:dim_length-1] {
> > > trace("Splitting ", i);
> > > input_chunks[i] = appSplitFile(input_data, split_dim, i+1);
> > > trace("Working furiously on ", @filename(input_chunks[i]));
> > > trace("Chunk number: ", i);
> > > processed_chunks[i] = input_chunks[i];
> > > }
> > >
> > >
> > > // Add all the chunks together
> > > iterate i {
> > > if( i == 0 ) {
> > > trace("Setting chunk 0 to ", @filename(output_chunks[0]));
> > > output_chunks[0] = processed_chunks[0];
> > > } else {
> > > trace("Adding ", i);
> > > output_chunks[i] = appAddFiles(processed_chunks[i],
> > > output_chunks[i-1]);
> > > }
> > > } until (i == dim_length-1);
> > >
> > >
> > > output_data = output_chunks[dim_length-1];
> > >
> > >
> > > // swift message
> > >
> > > RunID: 20121120-1538-mt4hjjd6
> > > Progress: time: Tue, 20 Nov 2012 15:38:03 -0800
> > > SwiftScript trace: Setting chunk 0 to ,
> > >
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0
> > > SwiftScript trace: filename of output_chunks[0] is,
> > >
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0
> > > SwiftScript trace: filename of processed_chunks[0] is,
> > >
> _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0
> > > Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1
> > > Finished
> > > successfully:1
> > > SwiftScript trace: Dim length = , 3.0
> > > SwiftScript trace: Splitting , 1.0
> > >
> > > SwiftScript trace: Chunk number: , 1.0
> > > SwiftScript trace: Chunk number: , 2.0
> > > SwiftScript trace: Working furiously on ,
> > >
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1
> > >
> > > SwiftScript trace: Splitting , 0.0
> > > SwiftScript trace: Chunk number: , 0.0
> > > SwiftScript trace: Splitting , 2.0
> > > SwiftScript trace: Working furiously on ,
> > >
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2
> > > SwiftScript trace: Working furiously on ,
> > >
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0
> > > Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1
> > > Stage
> > > in:1 Finished successfully:3
> > > Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1
> > > Finished successfully:4
> > > SwiftScript trace: Adding , 1.0
> > > SwiftScript trace: filename of output_chunks[i] is,
> > >
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1
> > > SwiftScript trace: filename of output_chunks[i-1] is,
> > >
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0
> > > SwiftScript trace: filename of processed_chunks[i] is,
> > >
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1
> > > SwiftScript trace: Adding , 2.0
> > > SwiftScript trace: filename of processed_chunks[i] is,
> > >
> _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2
> > > SwiftScript trace: filename of output_chunks[i] is,
> > >
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2
> > > SwiftScript trace: filename of output_chunks[i-1] is,
> > >
> _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1
> > > Execution failed:
> > > File not found:
> > >
> /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2
> > >
> > >
> > > From the trace we can see filename of output_chunks[2] is
> > > _concurrent/output_chunks- 33fce3ec-f16c-4430-b086-
> > > cc90f611a0a2--array//elt-2 (we also verified the file is there after
> > > the execution failure).
> > >
> > >
> > > Thank you.
> > >
> > > On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde < wilde at mcs.anl.gov >
> > > wrote:
> > >
> > >
> > > Hao,
> > >
> > > Are you sure you assigned a value to output_chunks[2]?
> > >
> > > Can you post the entire program, and the entire error message?
> > >
> > > Thanks,
> > >
> > > - Mike
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Hao Yang" < yanghao0614 at gmail.com >
> > > > To: swift-user at ci.uchicago.edu
> > > > Cc: mosastore at googlegroups.com
> > > > Sent: Tuesday, November 20, 2012 7:22:45 PM
> > > > Subject: [Swift-user] assign temporary file to mapper
> > > > Hi, all:
> > > >
> > > >
> > > > I have some problem assigning temporary file to some mapper
> > > > variable
> > > > (explained in the code below).
> > > >
> > > >
> > > > type RSFile;
> > > > string output_name;
> > > > output_name = @arg("out","out.rsf");
> > > > RSFile output_data <single_file_mapper; file=output_name>;
> > > > RSFile output_chunks[];
> > > > // some calculation to compute output_chunks[0], output_chunks[1],
> > > > output_chunks[2]
> > > > // at the end
> > > > output_data = output_chunks[2];
> > > >
> > > >
> > > >
> > > > When I try this script, swift reports " File output_chunks[2] not
> > > > found " error and failed. But the output_chunks[2] is still
> > > > available
> > > > in the directory generated by swift ( _ concurrent/output_chunks-
> > > > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error
> > >
> > > > message be expected?
> > > >
> > > >
> > > > Thank you.
> > > >
> > > > --
> > > > Best Regards,
> > > > Hao Yang
> > > >
> > > > The University of British Columbia
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Swift-user mailing list
> > > > Swift-user at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >
> > > --
> > > Michael Wilde
> > > Computation Institute, University of Chicago
> > > Mathematics and Computer Science Division
> > > Argonne National Laboratory
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Hao Yang
> > > Networked Systems Laboratory
> > > Electrical and Computer Engineering
> > > The University of British Columbia
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >
> >
> >
> >
> >
> > --
> > Best Regards,
> > Hao Yang
> > Networked Systems Laboratory
> > Electrical and Computer Engineering
> > The University of British Columbia
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>


-- 
Best Regards,
Hao Yang
Networked Systems Laboratory
Electrical and Computer Engineering
The University of British Columbia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20121130/57cd9ed4/attachment.html>