<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Justin,<div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I am a little confused by your response that cleaning up temporary files is not the responsibility of the Swift language. We did not </div><div>create the file 'wgt_files-935f5705-27ed-4a99-9420-441269bba3a0-36-4-0-array' Swift did. I certainly have not use for it. It was created </div><div>as part of the parallelization process. Consider the following bit of pseudo swift code</div><div><br></div><div><blockquote cite="mid:C29D99C4-40AC-4B77-9264-3DD9FFD59C31@ucar.edu" type="cite"></blockquote><span class="Apple-tab-span" style="white-space: pre; "> </span>foreach years {<font class="Apple-style-span" color="#144FAE"><br></font><blockquote cite="mid:C29D99C4-40AC-4B77-9264-3DD9FFD59C31@ucar.edu" type="cite"></blockquote><span class="Apple-tab-span" style="white-space: pre; "> </span>file wgt_files[];<br><blockquote cite="mid:C29D99C4-40AC-4B77-9264-3DD9FFD59C31@ucar.edu" type="cite"></blockquote><span class="Apple-tab-span" style="white-space: pre; "> </span>foreach month {<br><blockquote cite="mid:C29D99C4-40AC-4B77-9264-3DD9FFD59C31@ucar.edu" type="cite"></blockquote><span class="Apple-tab-span" style="white-space: pre; "> </span>wgt_files[] = DoSomething();<br><blockquote cite="mid:C29D99C4-40AC-4B77-9264-3DD9FFD59C31@ucar.edu" type="cite"></blockquote><span class="Apple-tab-span" style="white-space: pre; "> </span>}<span class="Apple-tab-span" style="white-space:pre"> </span><br><div><span class="Apple-tab-span" style="white-space: pre; "> </span>}</div><div><div><br></div></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>The 'wgt_files' is only in scope within the 'foreach years' loop. Once all iterations of 'foreach years' loop has completed, </div><div>I would expect the 'wgt_files' to be deleted once a variable/file goes out of scope. Isn't this really an issue of garbage collection </div><div>for the Swift language? </div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>While I do see how you could use the external variable to manage this all ourselves that would significantly complicate the </div><div>source code and remove much of the simple and elegant solution that Swift provides. </div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Matthew and I are concerned about this because of the impact this has on disk usage. For example our Swift script </div><div>requires temporary space of size 4x the input data. Our generated data is tiny, while the size of the _concurrent directory </div><div>is 2x the size of the input data. Now we want to execute the Swift script on ~30 TB of data. So just to enable parallel execution</div><div>with Swift would require an extra 120TB of disk space. I realize that parallel execution will consume more disk space but this seems </div><div>excessive.</div><div><br></div><div>Thanks,</div><div>John Dennis</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><br></div><div><br></div><div><div>On Aug 30, 2010, at 3:54 PM, Justin M Wozniak wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi Matthew<br><span class="Apple-tab-span" style="white-space:pre"> </span>Deleting files is out of the scope of the Swift language. You can of course remove them yourself in your scripts, and as long as Swift does not try to stage them out you should be fine.<br><span class="Apple-tab-span" style="white-space:pre"> </span>You may want to look at external variables as another way to approach this (manual 2.5). Using external variables you can manage the files in your scripts while maintaining the Swift progress model.<br><span class="Apple-tab-span" style="white-space:pre"> </span>Justin<br><br>On Fri, 27 Aug 2010, Matthew Woitaszek wrote:<br><blockquote type="cite">Good afternoon,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I'm working with a script that creates arrays of intermediate files<br></blockquote><blockquote type="cite">using the anonymous concurrent mapper, such as:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> file wgt_file[];<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">As I expect, all of these files get generated in the remote swift<br></blockquote><blockquote type="cite">temporary directory and are then returned to the _concurrent directory<br></blockquote><blockquote type="cite">on the host executing Swift. However, in this particular application,<br></blockquote><blockquote type="cite">they're then immediately consumed by a subsequent procedure and never<br></blockquote><blockquote type="cite">needed again.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Is there a way to configure Swift or the file mapper declaration to<br></blockquote><blockquote type="cite">delete these files after the remaining script "consumes" them? (That<br></blockquote><blockquote type="cite">is, after all procedures relying on them as inputs have been<br></blockquote><blockquote type="cite">executed?) Or can (should?) that be done manually?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">More speculatively, is there a way to keep files like these on the<br></blockquote><blockquote type="cite">execution host and not even bring them back to _concurrent? (With loss<br></blockquote><blockquote type="cite">of generality, I'm executing on a single site, and don't really ever<br></blockquote><blockquote type="cite">need the file locally, for restarts or staging to another site.)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Any advice about managing copies of large intermediate data files in<br></blockquote><blockquote type="cite">the Swift execution context would be appreciated!<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Matthew<br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Swift-user mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br></blockquote><blockquote type="cite"><a href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-user">http://mail.ci.uchicago.edu/mailman/listinfo/swift-user</a><br></blockquote><blockquote type="cite"><br></blockquote><br>-- <br>Justin M Wozniak<br>_______________________________________________<br>Swift-user mailing list<br><a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>http://mail.ci.uchicago.edu/mailman/listinfo/swift-user<br></div></blockquote></div><br></div></body></html>