<div>Hi, Michael: </div><div><br></div><div>The output_chunks[2] is assigned in the iterate via output_chunks[i] = appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the script code and the error message below). </div>
<div><br></div><div><b>// swift code </b></div><div>type RSFile; // rsf file</div><div>type StringFile; // File that stores strings</div><div><br></div><div>// APPs </div><div>// getNDims: Extract last dimension from rsf file</div>
<div>app (StringFile ndims) appGetNDims(RSFile input) {</div><div> sffiledims_script @filename(input) "1" stdout=@filename(ndims);</div><div>}</div><div><br></div><div>// getDims: Extract dimension list from rsf file</div>
<div>app (StringFile dims) appGetDims(RSFile input) {</div><div> sffiledims_script @filename(input) "2" stdout=@filename(dims);</div><div>}</div><div><br></div><div>// getWindow: Window a slice from rsf file</div>
<div>app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) {</div><div> sfwindow_script @filename(inp) @filename(out) dim slice;</div><div>}</div><div><br></div><div>// addFiles: adds inp1 and inp2 and returns out</div>
<div>app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) {</div><div> sfadd_script @filename(out) @filename(inp1) @filename(inp2);</div><div>}</div><div><br></div><div>// In-core variables</div><div>string input_name; // input name of the file</div>
<div>string output_name; // output name of the file</div><div>string last_dim; // Last dimension of file (default splitting dimension)</div><div>string split_dim; // Splitting dimension</div><div>string dims_list; // string of dimensions (unsplitted)</div>
<div>string dims_array[]; // Array of string of dimensions (splitted)</div><div>int dim_length; // length of splitted dimension</div><div><br></div><div>// File variables</div><div>// Setup input and output files</div><div>
RSFile input_data <single_file_mapper; file=input_name>;</div><div>RSFile output_data <single_file_mapper; file=output_name>;</div><div>StringFile ndims_file; // number of dimensions of the input file</div><div>
StringFile dims_file; // dimension array of the input file</div><div>RSFile input_chunks[]; // Splitted input</div><div>RSFile processed_chunks[]; // data chunks after parallel processing</div><div>RSFile output_chunks[]; // Temporary output array for reduction</div>
<div><br></div><div>// main part</div><div>// Define and parse input arguments</div><div>input_name = @arg("in","in.rsf");</div><div>output_name = @arg("out","out.rsf");</div><div><br>
</div><div>// extract dims and setup variables</div><div>ndims_file = appGetNDims(input_data);</div><div>dims_file = appGetDims(input_data);</div><div><br></div><div>// Define and parse split dimension and dimensions array</div>
<div>last_dim = readData(ndims_file);</div><div>split_dim = @arg("dim", last_dim);</div><div>dims_list = readData(dims_file);</div><div>dims_array = @strsplit(dims_list,",");</div><div>dim_length = @toint(dims_array[@toint(split_dim)-1]);</div>
<div>trace("Dim length = ", dim_length);</div><div><br></div><div>// Split the files parallelly</div><div>foreach i in [0:dim_length-1] {</div><div> trace("Splitting ", i);</div><div> input_chunks[i] = appSplitFile(input_data, split_dim, i+1);</div>
<div> trace("Working furiously on ", @filename(input_chunks[i]));</div><div> trace("Chunk number: ", i);</div><div> processed_chunks[i] = input_chunks[i];</div><div>}</div><div><br></div><div>
// Add all the chunks together</div><div>iterate i {</div><div> if( i == 0 ) {</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>trace("Setting chunk 0 to ", @filename(output_chunks[0]));</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>output_chunks[0] = processed_chunks[0];</div><div> } else {</div><div> <span class="Apple-tab-span" style="white-space:pre"> </span>trace("Adding ", i);</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>output_chunks[i] = appAddFiles(processed_chunks[i], output_chunks[i-1]);</div><div> }</div><div>} until (i == dim_length-1);</div><div><br></div><div>
output_data = output_chunks[dim_length-1];</div><div><br></div><div><b>// swift message</b> </div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
RunID: 20121120-1538-mt4hjjd6</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">Progress: time: Tue, 20 Nov 2012 15:38:03 -0800</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: Setting chunk 0 to , _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: filename of output_chunks[0] is, _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: filename of processed_chunks[0] is, _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1 Finished successfully:1</div><div class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: Dim length = , 3.0</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">SwiftScript trace: Splitting , 1.0</div><div class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
<div>SwiftScript trace: Chunk number: , 1.0</div><div>SwiftScript trace: Chunk number: , 2.0</div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: Working furiously on , _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1</div><div class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
<div>SwiftScript trace: Splitting , 0.0</div><div>SwiftScript trace: Chunk number: , 0.0</div></div><div class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: Splitting , 2.0</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">SwiftScript trace: Working furiously on , _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">SwiftScript trace: Working furiously on , _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1 Stage in:1 Finished successfully:3</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1 Finished successfully:4</div>
<div class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">SwiftScript trace: Adding , 1.0</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: filename of output_chunks[i] is, _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: filename of output_chunks[i-1] is, _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: filename of processed_chunks[i] is, _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1</div><div class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
SwiftScript trace: Adding , 2.0</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">SwiftScript trace: filename of processed_chunks[i] is, _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">SwiftScript trace: filename of output_chunks[i] is, _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">SwiftScript trace: filename of output_chunks[i-1] is, _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">Execution failed:</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
<span style="white-space:pre-wrap"> </span>File not found: /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2</div>
</div><div><br></div><div>From the trace we can see filename of output_chunks[2] is <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">_concurrent/output_chunks-</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">33fce3ec-f16c-4430-b086-</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">cc90f611a0a2--array//elt-2 (we also verified the file is there after the execution failure). </span></div>
<div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">Thank you. </span></div>
<br><div class="gmail_quote">On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hao,<br>
<br>
Are you sure you assigned a value to output_chunks[2]?<br>
<br>
Can you post the entire program, and the entire error message?<br>
<br>
Thanks,<br>
<br>
- Mike<br>
<div class="im"><br>
<br>
----- Original Message -----<br>
> From: "Hao Yang" <<a href="mailto:yanghao0614@gmail.com">yanghao0614@gmail.com</a>><br>
> To: <a href="mailto:swift-user@ci.uchicago.edu">swift-user@ci.uchicago.edu</a><br>
> Cc: <a href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a><br>
> Sent: Tuesday, November 20, 2012 7:22:45 PM<br>
> Subject: [Swift-user] assign temporary file to mapper<br>
> Hi, all:<br>
><br>
><br>
> I have some problem assigning temporary file to some mapper variable<br>
> (explained in the code below).<br>
><br>
><br>
> type RSFile;<br>
> string output_name;<br>
> output_name = @arg("out","out.rsf");<br>
> RSFile output_data <single_file_mapper; file=output_name>;<br>
> RSFile output_chunks[];<br>
> // some calculation to compute output_chunks[0], output_chunks[1],<br>
> output_chunks[2]<br>
> // at the end<br>
> output_data = output_chunks[2];<br>
><br>
><br>
><br>
> When I try this script, swift reports " File output_chunks[2] not<br>
> found " error and failed. But the output_chunks[2] is still available<br>
> in the directory generated by swift ( _ concurrent/output_chunks-<br>
</div>> ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error<br>
<div class="im">> message be expected?<br>
><br>
><br>
> Thank you.<br>
><br>
> --<br>
> Best Regards,<br>
> Hao Yang<br>
><br>
> The University of British Columbia<br>
><br>
><br>
><br>
><br>
><br>
</div>> _______________________________________________<br>
> Swift-user mailing list<br>
> <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Michael Wilde<br>
Computation Institute, University of Chicago<br>
Mathematics and Computer Science Division<br>
Argonne National Laboratory<br>
<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best Regards, <br>Hao Yang<div>Networked Systems Laboratory</div><div>Electrical and Computer Engineering<div>The University of British Columbia<br>
<div><div><br></div></div></div></div><br>