[Swift-user] Reduction trees
Bronevetsky, Greg
bronevetsky1 at llnl.gov
Sun May 18 17:16:33 CDT 2014
I need to implement a reduction three in Swift to aggregate the results of many individual runs. I've written the simple algorithm below, which works when all my runs correspond to numbers in a fixed range. However, in reality I have a regular or associative array of files produced by individual runs. How do I adapt the code below to this scenario? I don't see any way to iterate over subsets of array indexes/keys. Thanks!
Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com
type file;
app (file outFile) gen(string arg)
{
echo arg stdout=@filename(outFile);
}
app (file summaryFile) catBase(file inFiles[]) {
cat @filenames(inFiles) stdout=@filename(summaryFile);
}
// Concatenate the text of numbers in range [minR - maxR) (includes minR, not maxR)
(file summaryFile) catTree(int minR, int maxR, int radix, int level) {
file subFiles[];
if((maxR-minR) <= radix) {
//tracef("catTree: leaf: minR=%i, minR=%i\n", minR, maxR);
foreach b in [0: (maxR-minR)-1] {
//tracef("catTree: leaf: b=%i\n", b);
file curLeafFile <single_file_mapper; file=@strcat("file.", at toString(minR+b))>;
(curLeafFile) = gen(@toString(minR+b));
subFiles[b] = curLeafFile;
}
} else {
int size=maxR-minR;
//tracef("catTree: node: minR=%i, minR=%i\n", minR, maxR);
foreach b in [0: radix-1] {
file curNodeFile <single_file_mapper; file=@strcat("node.level_", at toString(level),".startVal_", at toString(minR+b))>;
int start = minR + (size*b)%/radix;
int end = minR + (size*(b+1))%/radix;
//tracef("catTree: node: b=%i [%i - %i]\n", b, start, end);
(curNodeFile) = catTree(start, end, radix, level+1);
subFiles[b] = curNodeFile;
}
}
//tracef("catBase: inFiles=%q\n", @filenames(subFiles));
(summaryFile) = catBase(subFiles);
}
file allFile <single_file_mapper; file="all">;
(allFile) = catTree(0, 31, 2, 0);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140518/d7236efa/attachment.html>
More information about the Swift-user
mailing list