[Swift-user] assign temporary file to mapper
Hao Yang
yanghao0614 at gmail.com
Tue Nov 20 22:33:15 CST 2012
Hi, Michael:
The output_chunks[2] is assigned in the iterate via output_chunks[i] =
appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the script
code and the error message below).
*// swift code *
type RSFile; // rsf file
type StringFile; // File that stores strings
// APPs
// getNDims: Extract last dimension from rsf file
app (StringFile ndims) appGetNDims(RSFile input) {
sffiledims_script @filename(input) "1" stdout=@filename(ndims);
}
// getDims: Extract dimension list from rsf file
app (StringFile dims) appGetDims(RSFile input) {
sffiledims_script @filename(input) "2" stdout=@filename(dims);
}
// getWindow: Window a slice from rsf file
app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) {
sfwindow_script @filename(inp) @filename(out) dim slice;
}
// addFiles: adds inp1 and inp2 and returns out
app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) {
sfadd_script @filename(out) @filename(inp1) @filename(inp2);
}
// In-core variables
string input_name; // input name of the file
string output_name; // output name of the file
string last_dim; // Last dimension of file (default splitting dimension)
string split_dim; // Splitting dimension
string dims_list; // string of dimensions (unsplitted)
string dims_array[]; // Array of string of dimensions (splitted)
int dim_length; // length of splitted dimension
// File variables
// Setup input and output files
RSFile input_data <single_file_mapper; file=input_name>;
RSFile output_data <single_file_mapper; file=output_name>;
StringFile ndims_file; // number of dimensions of the input file
StringFile dims_file; // dimension array of the input file
RSFile input_chunks[]; // Splitted input
RSFile processed_chunks[]; // data chunks after parallel processing
RSFile output_chunks[]; // Temporary output array for reduction
// main part
// Define and parse input arguments
input_name = @arg("in","in.rsf");
output_name = @arg("out","out.rsf");
// extract dims and setup variables
ndims_file = appGetNDims(input_data);
dims_file = appGetDims(input_data);
// Define and parse split dimension and dimensions array
last_dim = readData(ndims_file);
split_dim = @arg("dim", last_dim);
dims_list = readData(dims_file);
dims_array = @strsplit(dims_list,",");
dim_length = @toint(dims_array[@toint(split_dim)-1]);
trace("Dim length = ", dim_length);
// Split the files parallelly
foreach i in [0:dim_length-1] {
trace("Splitting ", i);
input_chunks[i] = appSplitFile(input_data, split_dim, i+1);
trace("Working furiously on ", @filename(input_chunks[i]));
trace("Chunk number: ", i);
processed_chunks[i] = input_chunks[i];
}
// Add all the chunks together
iterate i {
if( i == 0 ) {
trace("Setting chunk 0 to ", @filename(output_chunks[0]));
output_chunks[0] = processed_chunks[0];
} else {
trace("Adding ", i);
output_chunks[i] = appAddFiles(processed_chunks[i], output_chunks[i-1]);
}
} until (i == dim_length-1);
output_data = output_chunks[dim_length-1];
*// swift message*
RunID: 20121120-1538-mt4hjjd6
Progress: time: Tue, 20 Nov 2012 15:38:03 -0800
SwiftScript trace: Setting chunk 0 to ,
_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0
SwiftScript trace: filename of output_chunks[0] is,
_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0
SwiftScript trace: filename of processed_chunks[0] is,
_concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0
Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1 Finished
successfully:1
SwiftScript trace: Dim length = , 3.0
SwiftScript trace: Splitting , 1.0
SwiftScript trace: Chunk number: , 1.0
SwiftScript trace: Chunk number: , 2.0
SwiftScript trace: Working furiously on ,
_concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1
SwiftScript trace: Splitting , 0.0
SwiftScript trace: Chunk number: , 0.0
SwiftScript trace: Splitting , 2.0
SwiftScript trace: Working furiously on ,
_concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2
SwiftScript trace: Working furiously on ,
_concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0
Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1 Stage
in:1 Finished successfully:3
Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1
Finished successfully:4
SwiftScript trace: Adding , 1.0
SwiftScript trace: filename of output_chunks[i] is,
_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1
SwiftScript trace: filename of output_chunks[i-1] is,
_concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0
SwiftScript trace: filename of processed_chunks[i] is,
_concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1
SwiftScript trace: Adding , 2.0
SwiftScript trace: filename of processed_chunks[i] is,
_concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2
SwiftScript trace: filename of output_chunks[i] is,
_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2
SwiftScript trace: filename of output_chunks[i-1] is,
_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1
Execution failed:
File not found:
/users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2
>From the trace we can see filename of output_chunks[2] is
_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2
(we also verified the file is there after the execution failure).
Thank you.
On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:
> Hao,
>
> Are you sure you assigned a value to output_chunks[2]?
>
> Can you post the entire program, and the entire error message?
>
> Thanks,
>
> - Mike
>
>
> ----- Original Message -----
> > From: "Hao Yang" <yanghao0614 at gmail.com>
> > To: swift-user at ci.uchicago.edu
> > Cc: mosastore at googlegroups.com
> > Sent: Tuesday, November 20, 2012 7:22:45 PM
> > Subject: [Swift-user] assign temporary file to mapper
> > Hi, all:
> >
> >
> > I have some problem assigning temporary file to some mapper variable
> > (explained in the code below).
> >
> >
> > type RSFile;
> > string output_name;
> > output_name = @arg("out","out.rsf");
> > RSFile output_data <single_file_mapper; file=output_name>;
> > RSFile output_chunks[];
> > // some calculation to compute output_chunks[0], output_chunks[1],
> > output_chunks[2]
> > // at the end
> > output_data = output_chunks[2];
> >
> >
> >
> > When I try this script, swift reports " File output_chunks[2] not
> > found " error and failed. But the output_chunks[2] is still available
> > in the directory generated by swift ( _ concurrent/output_chunks-
> > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error
> > message be expected?
> >
> >
> > Thank you.
> >
> > --
> > Best Regards,
> > Hao Yang
> >
> > The University of British Columbia
> >
> >
> >
> >
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>
--
Best Regards,
Hao Yang
Networked Systems Laboratory
Electrical and Computer Engineering
The University of British Columbia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20121120/ee59510b/attachment.html>
More information about the Swift-user
mailing list