[Swift-user] using dual filesys_mappers

[Robin Weiss]

After a bit more playing, i have this working now.  I also realized that 'log' is really an output and shouldn't be passed into my app as an input (doh!).  The way I'm doing it now is as follows:


app (clusFile clus, logFile log) processData (dataFile data, metaFile meta){


boot "-log" @log "-results" @clus "-meta" @meta "-data" @data "-kmin" kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations" maxiterations "-maxtries" maxtries;

}


metaFile metaFiles[] <filesys_mapper;location="in",pattern="*.meta">;

dataFile dataFiles[] <filesys_mapper;location="in",pattern="*.dat">;


foreach f,i in dataFiles {


clusFile clus<regexp_mapper;source=@f,match="(in)/(.*)dat",transform="out/\\2clus">;

logFile log<regexp_mapper;source=@f,match="(in)/(.*)dat",transform="out/\\2log">;


(clus,log) = processData(f, metaFiles[i]);


}



This script is now mapping the .meta and .dat files from the 'in' directory with the pattern option of filesys_mapper.  Perhaps i'm missing something about what the suffix option does.


Cheers,

Robin


--
Robin M. Weiss
Research Programmer
Research Computing Center
The University of Chicago
6030 S. Ellis Ave., Suite 289C
Chicago, IL 60637
robinweiss at uchicago.edu<mailto:labello at uchicago.edu>
773.702.9030

From: Robin Weiss <robinweiss at uchicago.edu<mailto:robinweiss at uchicago.edu>>
Date: Mon, 16 Jul 2012 10:43:29 -0500
To: <swift-user at ci.uchicago.edu<mailto:swift-user at ci.uchicago.edu>>
Subject: using dual filesys_mappers


Hello Swifters,


I have a question about using two filesys_mappers and the foreach construct.  I have attached the offending .swift script I am working with for reference.  Here's the gist of what i'm trying to do and what appears to be happening instead:


I have a program called 'boot' that takes as command line arguments 4 file paths (log, out, data, and meta), and a number of other params (all numerical and seem to be getting passed in correctly, so no worries there).  'log' and 'out' are output files and 'data' and 'mata' are input files (located in directories called 'out' and 'in' respectively).  The problem i'm having is with getting the correct values for 'data' and 'meta' passed in to my app call.  I have an app section called processData that invokes boot.  I will be assuming the the directory 'in' contains identically named data and meta files that differ only in their suffix ('.dat' or '.meta', respectively).  This may or may not be safe, but for now it is fine and I'll cross that bridge when I come to it.  Here's the relevant snippet from my script:



app (clusFile out) processData (dataFile data, metaFile meta, logFile log){


        boot "-log" @log "-results" @out "-meta" @meta "-data" @data "-kmin" kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations" maxiterations "-maxtries" maxtries;


}


dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;

metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;


foreach f,i in dataFiles {


        clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;

        logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;

        o = processData(f, metaFiles[i], l);


}


this configuration causes processData to be invoked as:


out/clusFile.0.clus = processData(dataFile0.dat, dataFile0.dat, out/logFile.0.log);


if i switch around the oder of the filesys_mappers so that the snippet reads:


metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;

dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;


foreach f,i in dataFiles {


        clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;

        logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;

        o = processData(f, metaFiles[i], l);


}


the app invocation is called as:


out/clusFile.o.clus = processData(dataFile0.meta, dataFile0.meta, out/logFile.0.log);


I guess the real question is this: what is the most appropriate way in swift to pass into a app invocation two corresponding input files?  Ideally, it would be something like 'foreach file1,file2,i in inputFiles[][] { … } but that doesn't really make too much sense either.


Anyway, any ideas would be appreciated.


Cheers,

Robin


--
Robin M. Weiss
Research Programmer
Research Computing Center
The University of Chicago
6030 S. Ellis Ave., Suite 289C
Chicago, IL 60637
robinweiss at uchicago.edu<mailto:labello at uchicago.edu>
773.702.9030
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120716/28ef5d71/attachment.html>