[Swift-user] [Swift-devel] Multiple output files

[Michael Wilde]

Jonathan, I'd need to test this, but my suspicion is that specifying 
location only changes the directory that the mapper should look in.

But you also need to give it a prefix and/or a suffix - else it won't 
know what you want it to map *within* that drectory.

My guess is that you had a mapper with "prefix=" that worked, and you 
just changed "prefix" to "location"?

Also - please see Mihael's post to swift-devel in the past hour. He has 
committed the enhancement to dynamically map an output array of files on 
the compute node, as you requested and we discussed.  I think you'll 
have to build a swift from trunk source to try this, but thats only an 
svn co and a few commands (see Download at swift-lang.org).

-- 
Mike

On 3/27/14, 5:02 PM, Jonathan Ozik wrote:
> Mike,
>
> I've been making a good amount of progress and I'm able to 
> successfully run a number of concurrent instances locally.
>
> I had a question about the filesys_mapper, and how it might (or might 
> not) be able to handle nested folders. If I point it to a "location" 
> like this:
> file data[] <filesys_mapper;location="data">;
> I get a file not found error when it is used as an input (presumably 
> because that is when the mapper gets resolved) if there is a nested 
> folder within that location.
> data/
> temp.txt
> insideData/
> temp1.txt
>
> Is this expected behavior? If so, is there a workaround to gather all 
> files under a folder as input?
>
> Jonathan
>
> On Mar 22, 2014, at 11:31 AM, Michael Wilde <wilde at anl.gov 
> <mailto:wilde at anl.gov>> wrote:
>
>> This is now filed as enhancement bug 1225 and assigned to you, 
>> Mihael.  The description has been revised as you suggested, to 
>> propose that the feature be first done using simple_mapper rather 
>> than a new mapper.
>>
>> - Mike
>>
>> On 3/21/14, 8:39 AM, Michael Wilde wrote:
>>> Mihael, All,
>>>
>>> I'd like to propose a Swift/K feature to provide a reasonable 
>>> solution to this very common need for an app to return a dynamically 
>>> determined set of files.
>>>
>>> file dynarry[ ] <runtime; prefix="myoutput.", suffix=".out", 
>>> indexes="int">;
>>>
>>> dynarry = myApp(myArgs);
>>>
>>> The "runtime" mapper should initially have the same arguments and 
>>> semantics (roughly) as simple_mapper, except for two new arguments:
>>>
>>>  "indexes" which determines how the matched file names will be 
>>> indexed in the returned array
>>> "int" | "string" | "sequential"
>>> sequential: return the matched files as consecutive integer indices 
>>> starting with 0
>>> int: expect the filename component between prefix and suffix to be 
>>> convertible to an integer, and use that as the index
>>> eg myfile.012.out and myfile.204.out will return an array with the 
>>> mapped files at indices 12 and 204.
>>> string: similar to int but return a string-indexed associative array.
>>> "sequential" is simplest and should be the default.
>>>
>>> "paths" which determines if the match names will be absolute or 
>>> relative to the job dir
>>> paths="relative" | "absolute"
>>> (may not be needed if this can be determined uniquely based on the 
>>> location argument.
>>>
>>> swiftwrap will allow array variables mapped in this manner to have 
>>> any number of files, including zero. I.e. "runtime-mapped" files 
>>> should not be listed in the expected output list for an app 
>>> invocation. Its up to the users app to ensure that some files match 
>>> the pattern. An additional arg could set e.g. minfiles and/or 
>>> maxfiles, in which case the wrapper code needs to validate the count 
>>> of files matched and returned, but not their exact names.
>>>
>>> We can call this mapper "experimental" until we validate its 
>>> usability and suitability as a permanent feature. But as we hope to 
>>> revise the entire mapper family and semantics, in a sense all 
>>> mappers are subject to change.
>>>
>>> Mihael, is the definition sound, and how long would it take you to 
>>> develop it?
>>>
>>> Thanks,
>>>
>>> - Mike
>>>
>>
>> -- 
>> Michael Wilde
>> Mathematics and Computer Science          Computation Institute
>> Argonne National Laboratory               The University of Chicago
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140327/d55a7b35/attachment.html>