<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<div>
<div>
<div>So it turns out this was a PEBKAC issue. While I think I know why swift is doing what it's doing, this behavior is a little tricky to diagnose and I was totally looking at the problem but not seeing it. Let me explain.</div>
<div><br>
</div>
<div>Here is the offending swift script and the output from running it. The directory 'in' contains a number of .dat files and corresponding .meta files (name differs only in extension). The expected output in each outFile file should be "-meta fileX.meta
-dat fileX.dat", "-meta fileY.meta -dat fileY.dat", etc.</div>
<div><br>
</div>
<div><br>
</div>
<div>type datFile;</div>
<div>type metaFile;</div>
<div>type outFile;</div>
<div><br>
</div>
<div>app (outFile out) testApp (datFile dat, metaFile meta){</div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>echo "-meta" @meta "-dat" @dat stdout=@out;</div>
<div><br>
</div>
<div>}</div>
<div><br>
</div>
<div>datFile datFiles[] <filesys_mapper;location="in",sufix="dat">;</div>
<div>metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;</div>
<div><br>
</div>
<div>foreach f,i in datFiles {</div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>outFile out <concurrent_mapper;prefix="out", suffix=".txt">;</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span></div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>out = testApp(f, metaFiles[i]);</div>
<div> </div>
<div>}</div>
<div><br>
</div>
<div><br>
</div>
<div>[robinweiss@midway037 bad_swift]$ pwd</div>
<div>/home/robinweiss/bad_swift</div>
<div>[robinweiss@midway037 bad_swift]$ cd in</div>
<div>[robinweiss@midway037 in]$ ls</div>
<div>file0.dat file0.meta file1.dat file1.meta file2.dat file2.meta file3.dat file3.meta</div>
<div>[robinweiss@midway037 in]$ cd ..</div>
<div>[robinweiss@midway037 bad_swift]$ ./runLocal.sh </div>
<div>Swift 0.93 swift-r5483 cog-r3339</div>
<div><br>
</div>
<div>RunID: 20120720-1623-cpgd5xr9</div>
<div> (input): found 8 files</div>
<div> (input): found 8 files</div>
<div>Progress: time: Fri, 20 Jul 2012 16:23:48 +0000 Initializing:1</div>
<div>Final status: time: Fri, 20 Jul 2012 16:23:49 +0000 Finished successfully:8</div>
<div>[robinweiss@midway037 bad_swift]$ cd _concurrent/</div>
<div>[robinweiss@midway037 _concurrent]$ ls</div>
<div>out-4-0.txt out-4-1.txt out-4-2.txt out-4-3.txt out-4-4.txt out-4-5.txt out-4-6.txt out-4-7.txt</div>
<div>[robinweiss@midway037 _concurrent]$ cat *</div>
<div>-meta in/file0.meta -dat in/file0.meta</div>
<div>-meta in/file0.dat -dat in/file0.dat</div>
<div>-meta in/file1.meta -dat in/file1.meta</div>
<div>-meta in/file1.dat -dat in/file1.dat</div>
<div>-meta in/file2.dat -dat in/file2.dat</div>
<div>-meta in/file3.dat -dat in/file3.dat</div>
<div>-meta in/file3.meta -dat in/file3.meta</div>
<div>-meta in/file2.meta -dat in/file2.meta</div>
<div><br>
</div>
<div><br>
</div>
<div>Can you spot the problem in the script? Hint: "suffix" has two f's, not one. Because the parameter 'sufix' is meaningless to filesys_mapper it ends up just mapping everything in 'location'. It would seem mappers will allow you to put in pretty much
any garbage you want inside the < >'s so long as its an assignment. For example:</div>
<div><br>
</div>
<div>datFile datFiles[] <filesys_mapper;location="in",suffix="dat",foo="bar",biz="baz",garbage="more garbage">;</div>
<div><br>
</div>
<div>works just fine also.</div>
<div><br>
</div>
<div><br>
</div>
<div>And now for the good run:</div>
<div><br>
</div>
<div>type datFile;</div>
<div>type metaFile;</div>
<div>type outFile;</div>
<div><br>
</div>
<div>app (outFile out) testApp (datFile dat, metaFile meta){</div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>echo "-meta" @meta "-dat" @dat stdout=@out;</div>
<div><br>
</div>
<div>}</div>
<div><br>
</div>
<div>datFile datFiles[] <filesys_mapper;location="in",suffix="dat">;</div>
<div>metaFile metaFiles[] <filesys_mapper;location="in",suffix="meta">;</div>
<div><br>
</div>
<div>foreach f,i in datFiles {</div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>outFile out <concurrent_mapper;prefix="out", suffix=".txt">;</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span></div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>out = testApp(f, metaFiles[i]);</div>
<div> </div>
<div>}</div>
<div><br>
</div>
<div>[robinweiss@midway037 bad_swift]$ pwd</div>
<div>/home/robinweiss/bad_swift</div>
<div>[robinweiss@midway037 bad_swift]$ cd in</div>
<div>[robinweiss@midway037 in]$ ls</div>
<div>file0.dat file0.meta file1.dat file1.meta file2.dat file2.meta file3.dat file3.meta</div>
<div>[robinweiss@midway037 in]$ cd ..</div>
<div>[robinweiss@midway037 bad_swift]$ ./runLocal.sh </div>
<div>Swift 0.93 swift-r5483 cog-r3339</div>
<div><br>
</div>
<div>RunID: 20120720-1626-ou1juqf5</div>
<div> (input): found 4 files</div>
<div> (input): found 4 files</div>
<div>Progress: time: Fri, 20 Jul 2012 16:26:51 +0000 Initializing:1</div>
<div>Final status: time: Fri, 20 Jul 2012 16:26:51 +0000 Finished successfully:4</div>
<div>[robinweiss@midway037 bad_swift]$ cd _concurrent/</div>
<div>[robinweiss@midway037 _concurrent]$ ls</div>
<div>out-4-0.txt out-4-1.txt out-4-2.txt out-4-3.txt</div>
<div>[robinweiss@midway037 _concurrent]$ cat *</div>
<div>-meta in/file0.meta -dat in/file0.dat</div>
<div>-meta in/file1.meta -dat in/file1.dat</div>
<div>-meta in/file3.meta -dat in/file2.dat</div>
<div>-meta in/file2.meta -dat in/file3.dat</div>
<div>[robinweiss@midway037 _concurrent]$ </div>
<div><br>
</div>
<div><br>
</div>
<div>So like I said, it turns out this whole issue boils down to a typo. I would suggest that swift should throw up some sort of warning if you pass something to one of the pre-defined mappers that is not defined as one of its parameters. I would have expected
swift to complain that "sufix" is an unknown parameter of filesys_mapper.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Robin</div>
<div><br>
</div>
</div>
<div>
<div>
<div>-- </div>
<div>
<div>Robin M. Weiss</div>
<div>Research Programmer</div>
<div>Research Computing Center</div>
<div>The University of Chicago</div>
<div>6030 S. Ellis Ave., Suite 289C</div>
<div>Chicago, IL 60637</div>
<div><b><a href="mailto:labello@uchicago.edu">robinweiss@uchicago.edu</a></b></div>
<div>773.702.9030</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Robin Weiss <<a href="mailto:robinweiss@uchicago.edu">robinweiss@uchicago.edu</a>><br>
<span style="font-weight:bold">Date: </span>Mon, 16 Jul 2012 10:43:29 -0500<br>
<span style="font-weight:bold">To: </span><<a href="mailto:swift-user@ci.uchicago.edu">swift-user@ci.uchicago.edu</a>><br>
<span style="font-weight:bold">Subject: </span>using dual filesys_mappers<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<div>
<div>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">Hello Swifters,</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">I have a question about using two filesys_mappers and the foreach construct. I have attached the offending .swift script I am working with for reference. Here's the gist of what i'm trying
to do and what appears to be happening instead: </p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">I have a program called 'boot' that takes as command line arguments 4 file paths (log, out, data, and meta), and a number of other params (all numerical and seem to be getting passed in correctly,
so no worries there). 'log' and 'out' are output files and 'data' and 'mata' are input files (located in directories called 'out' and 'in' respectively). The problem i'm having is with getting the correct values for 'data' and 'meta' passed in to my app
call. I have an app section called processData that invokes boot. I will be assuming the the directory 'in' contains identically named data and meta files that differ only in their suffix ('.dat' or '.meta', respectively). This may or may not be safe, but
for now it is fine and I'll cross that bridge when I come to it. Here's the relevant snippet from my script:</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">app (clusFile out) processData (dataFile data, metaFile meta, logFile log){</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> boot "-log" @log "-results" @out "-meta" @meta "-data" @data "-kmin" kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations" maxiterations "-maxtries" maxtries;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">}</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">foreach f,i in dataFiles {</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> o = processData(f, metaFiles[i], l);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">}</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">this configuration causes processData to be invoked as: </p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">out/clusFile.0.clus = processData(dataFile0.dat, dataFile0.dat, out/logFile.0.log);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">if i switch around the oder of the filesys_mappers so that the snippet reads:</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">foreach f,i in dataFiles {</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> o = processData(f, metaFiles[i], l);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">}</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">the app invocation is called as:</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">out/clusFile.o.clus = processData(dataFile0.meta, dataFile0.meta, out/logFile.0.log);</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">I guess the real question is this: what is the most appropriate way in swift to pass into a app invocation two corresponding input files? Ideally, it would be something like 'foreach file1,file2,i
in inputFiles[][] { … } but that doesn't really make too much sense either.</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">Anyway, any ideas would be appreciated.</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">Cheers,</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">Robin</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">
<br>
</p>
</div>
<div>
<div>
<div>-- </div>
<div>
<div>Robin M. Weiss</div>
<div>Research Programmer</div>
<div>Research Computing Center</div>
<div>The University of Chicago</div>
<div>6030 S. Ellis Ave., Suite 289C</div>
<div>Chicago, IL 60637</div>
<div><b><a href="mailto:labello@uchicago.edu">robinweiss@uchicago.edu</a></b></div>
<div>773.702.9030</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span>
</body>
</html>