[Swift-user] Reduction trees

Bronevetsky, Greg bronevetsky1 at llnl.gov
Sun May 18 17:16:33 CDT 2014


I need to implement a reduction three in Swift to aggregate the results of many individual runs. I've written the simple algorithm below, which works when all my runs correspond to numbers in a fixed range. However, in reality I have a regular or associative array of files produced by individual runs.  How do I adapt the code below to this scenario? I don't see any way to iterate over subsets of array indexes/keys. Thanks!

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com


type file;

app (file outFile) gen(string arg)
{
  echo arg stdout=@filename(outFile);
}

app (file summaryFile) catBase(file inFiles[]) {
  cat @filenames(inFiles) stdout=@filename(summaryFile);
}

// Concatenate the text of numbers in range [minR - maxR) (includes minR, not maxR)
(file summaryFile) catTree(int minR, int maxR, int radix, int level) {
  file subFiles[];
 if((maxR-minR) <= radix) {
    //tracef("catTree: leaf: minR=%i, minR=%i\n", minR, maxR);
    foreach b in [0: (maxR-minR)-1] {
      //tracef("catTree: leaf: b=%i\n", b);
      file curLeafFile <single_file_mapper; file=@strcat("file.", at toString(minR+b))>;
      (curLeafFile) = gen(@toString(minR+b));
      subFiles[b] = curLeafFile;
    }
  } else {
    int size=maxR-minR;
    //tracef("catTree: node: minR=%i, minR=%i\n", minR, maxR);
    foreach b in [0: radix-1] {
      file curNodeFile <single_file_mapper; file=@strcat("node.level_", at toString(level),".startVal_", at toString(minR+b))>;
      int start = minR + (size*b)%/radix;
      int end   = minR + (size*(b+1))%/radix;
      //tracef("catTree: node: b=%i [%i - %i]\n", b, start, end);
      (curNodeFile) = catTree(start, end, radix, level+1);
      subFiles[b] = curNodeFile;
    }
  }
  //tracef("catBase: inFiles=%q\n", @filenames(subFiles));
  (summaryFile) = catBase(subFiles);
}


file allFile <single_file_mapper; file="all">;
(allFile) = catTree(0, 31, 2, 0);

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140518/d7236efa/attachment.html>


More information about the Swift-user mailing list