[Swift-user] Reduction trees

[Bronevetsky, Greg]

What I mean is that I have an array of files computed by my individual tasks. To do a reduction tree on them I need to create a sub-array of the first 100, the second 100, etc. so that each sub-array can be merged independently. In my code example below I skip this step and simply generate a region of numbers (minR, maxR) for each node in my reduction tree and then do stuff with the numbers. Now I need for these numbers to correspond to indexes in an array of files.

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov
http://greg.bronevetsky.com

-----Original Message-----
From: Mihael Hategan [mailto:hategan at mcs.anl.gov] 
Sent: Sunday, May 18, 2014 5:05 PM
To: Bronevetsky, Greg
Cc: swift-user at ci.uchicago.edu
Subject: Re: [Swift-user] Reduction trees

Hi,

Can you be more specific about what you mean by "subsets of keys" below?
Specifically, how are these sub sets defined?

Mihael

On Sun, 2014-05-18 at 22:16 +0000, Bronevetsky, Greg wrote:
> I need to implement a reduction three in Swift to aggregate the 
> results of many individual runs. I've written the simple algorithm 
> below, which works when all my runs correspond to numbers in a fixed 
> range. However, in reality I have a regular or associative array of 
> files produced by individual runs.  How do I adapt the code below to 
> this scenario? I don't see any way to iterate over subsets of array 
> indexes/keys. Thanks!
> 
> Greg Bronevetsky
> Lawrence Livermore National Lab
> (925) 424-5756
> bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
> http://greg.bronevetsky.com
> 
> 
> type file;
> 
> app (file outFile) gen(string arg)
> {
>   echo arg stdout=@filename(outFile);
> }
> 
> app (file summaryFile) catBase(file inFiles[]) {
>   cat @filenames(inFiles) stdout=@filename(summaryFile); }
> 
> // Concatenate the text of numbers in range [minR - maxR) (includes 
> minR, not maxR) (file summaryFile) catTree(int minR, int maxR, int radix, int level) {
>   file subFiles[];
>  if((maxR-minR) <= radix) {
>     //tracef("catTree: leaf: minR=%i, minR=%i\n", minR, maxR);
>     foreach b in [0: (maxR-minR)-1] {
>       //tracef("catTree: leaf: b=%i\n", b);
>       file curLeafFile <single_file_mapper; file=@strcat("file.", at toString(minR+b))>;
>       (curLeafFile) = gen(@toString(minR+b));
>       subFiles[b] = curLeafFile;
>     }
>   } else {
>     int size=maxR-minR;
>     //tracef("catTree: node: minR=%i, minR=%i\n", minR, maxR);
>     foreach b in [0: radix-1] {
>       file curNodeFile <single_file_mapper; file=@strcat("node.level_", at toString(level),".startVal_", at toString(minR+b))>;
>       int start = minR + (size*b)%/radix;
>       int end   = minR + (size*(b+1))%/radix;
>       //tracef("catTree: node: b=%i [%i - %i]\n", b, start, end);
>       (curNodeFile) = catTree(start, end, radix, level+1);
>       subFiles[b] = curNodeFile;
>     }
>   }
>   //tracef("catBase: inFiles=%q\n", @filenames(subFiles));
>   (summaryFile) = catBase(subFiles);
> }
> 
> 
> file allFile <single_file_mapper; file="all">;
> (allFile) = catTree(0, 31, 2, 0);
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user