[Swift-user] using dual filesys_mappers
Michael Wilde
wilde at mcs.anl.gov
Wed Jul 18 16:48:43 CDT 2012
Robin, here's an example of using the suffix option for the filesys_mapper:
login2$ ls *.pcap
pc00.pcap pc01.pcap pc02.pcap pc03.pcap pcaa.pcap pczz.pcap
login2$ cat filesysmapper.swift
type pcapfile;
pcapfile pcapfiles[]<filesys_mapper; prefix="pc", suffix=".pcap">;
foreach pf,i in pcapfiles {
# trace(i, at filename(pf));
}
iterate i {
trace(i, @filename(pcapfiles[i]));
} until (i==@length(pcapfiles));
login2$ swift filesysmapper.swift
no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml
Swift trunk swift-r5781 cog-r3368 (cog modified locally)
RunID: 20120718-1646-fm7zxfka
(input): found 6 files
Progress: time: Wed, 18 Jul 2012 16:46:12 -0500
SwiftScript trace: 0, pczz.pcap
SwiftScript trace: 1, pc00.pcap
SwiftScript trace: 2, pc01.pcap
SwiftScript trace: 3, pc03.pcap
SwiftScript trace: 4, pcaa.pcap
SwiftScript trace: 5, pc02.pcap
Final status: Wed, 18 Jul 2012 16:46:12 -0500
login2$
I need to go back to your prior email to see if you discovered a problem with it.
- Mike
----- Original Message -----
> From: "Robin Weiss" <robinweiss at uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Monday, July 16, 2012 12:34:23 PM
> Subject: Re: [Swift-user] using dual filesys_mappers
> After a bit more playing, i have this working now. I also realized
> that 'log' is really an output and shouldn't be passed into my app as
> an input (doh!). The way I'm doing it now is as follows:
>
>
>
>
> app (clusFile clus, logFile log) processData (dataFile data, metaFile
> meta){
>
>
>
>
> boot "-log" @log "-results" @clus "-meta" @meta "-data" @data "-kmin"
> kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations"
> maxiterations "-maxtries" maxtries;
>
>
>
> }
>
>
>
>
> metaFile metaFiles[] <filesys_mapper;location="in",pattern="*.meta">;
>
> dataFile dataFiles[] <filesys_mapper;location="in",pattern="*.dat">;
>
>
>
>
> foreach f,i in dataFiles {
>
>
>
>
> clusFile
> clus<regexp_mapper;source=@f,match="(in)/(.*)dat",transform="out/\\2clus">;
>
> logFile
> log<regexp_mapper;source=@f,match="(in)/(.*)dat",transform="out/\\2log">;
>
>
>
>
> (clus,log) = processData(f, metaFiles[i]);
>
>
>
>
> }
>
>
>
>
>
>
>
> This script is now mapping the .meta and .dat files from the 'in'
> directory with the pattern option of filesys_mapper. Perhaps i'm
> missing something about what the suffix option does.
>
>
>
>
> Cheers,
>
> Robin
>
>
>
>
>
> --
>
> Robin M. Weiss
> Research Programmer
> Research Computing Center
> The University of Chicago
> 6030 S. Ellis Ave., Suite 289C
> Chicago, IL 60637
> robinweiss at uchicago.edu
> 773.702.9030
>
>
> From: Robin Weiss < robinweiss at uchicago.edu >
> Date: Mon, 16 Jul 2012 10:43:29 -0500
> To: < swift-user at ci.uchicago.edu >
> Subject: using dual filesys_mappers
>
>
>
>
>
>
>
>
>
> Hello Swifters,
>
>
>
>
> I have a question about using two filesys_mappers and the foreach
> construct. I have attached the offending .swift script I am working
> with for reference. Here's the gist of what i'm trying to do and what
> appears to be happening instead:
>
>
>
>
> I have a program called 'boot' that takes as command line arguments 4
> file paths (log, out, data, and meta), and a number of other params
> (all numerical and seem to be getting passed in correctly, so no
> worries there). 'log' and 'out' are output files and 'data' and 'mata'
> are input files (located in directories called 'out' and 'in'
> respectively). The problem i'm having is with getting the correct
> values for 'data' and 'meta' passed in to my app call. I have an app
> section called processData that invokes boot. I will be assuming the
> the directory 'in' contains identically named data and meta files that
> differ only in their suffix ('.dat' or '.meta', respectively). This
> may or may not be safe, but for now it is fine and I'll cross that
> bridge when I come to it. Here's the relevant snippet from my script:
>
>
>
>
>
>
>
> app (clusFile out) processData (dataFile data, metaFile meta, logFile
> log){
>
>
>
>
> boot "-log" @log "-results" @out "-meta" @meta "-data" @data "-kmin"
> kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations"
> maxiterations "-maxtries" maxtries;
>
>
>
>
> }
>
>
>
>
> dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;
>
> metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;
>
>
>
>
> foreach f,i in dataFiles {
>
>
>
>
> clusFile o<single_file_mapper; location="out",
> file=@strcat("out/clusFile.",i,".clus")>;
>
> logFile l<single_file_mapper; location="out",
> file=@strcat("out/logFile.",i,".log")>;
>
> o = processData(f, metaFiles[i], l);
>
>
>
>
> }
>
>
>
>
> this configuration causes processData to be invoked as:
>
>
>
>
> out/clusFile.0.clus = processData(dataFile0.dat, dataFile0.dat,
> out/logFile.0.log);
>
>
>
>
> if i switch around the oder of the filesys_mappers so that the snippet
> reads:
>
>
>
>
> metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;
>
> dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;
>
>
>
>
> foreach f,i in dataFiles {
>
>
>
>
> clusFile o<single_file_mapper; location="out",
> file=@strcat("out/clusFile.",i,".clus")>;
>
> logFile l<single_file_mapper; location="out",
> file=@strcat("out/logFile.",i,".log")>;
>
> o = processData(f, metaFiles[i], l);
>
>
>
>
> }
>
>
>
>
> the app invocation is called as:
>
>
>
>
> out/clusFile.o.clus = processData(dataFile0.meta, dataFile0.meta,
> out/logFile.0.log);
>
>
>
>
> I guess the real question is this: what is the most appropriate way in
> swift to pass into a app invocation two corresponding input files?
> Ideally, it would be something like 'foreach file1,file2,i in
> inputFiles[][] { … } but that doesn't really make too much sense
> either.
>
>
>
>
> Anyway, any ideas would be appreciated.
>
>
>
>
> Cheers,
>
> Robin
>
>
>
>
>
> --
>
> Robin M. Weiss
> Research Programmer
> Research Computing Center
> The University of Chicago
> 6030 S. Ellis Ave., Suite 289C
> Chicago, IL 60637
> robinweiss at uchicago.edu
> 773.702.9030
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
--
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory
More information about the Swift-user
mailing list