[Swift-devel] Multiple output files

[Michael Wilde]

Resending to cc'ing Jonathan.

Lets try to quickly converge on a spec to implement, and send it to 
swift-user for comments.

It also occurs to me that this feature will interact with the semantics 
of "implicitly-direct" data management conventions that we have long 
discussed. So we should keep an eye out for how to solve both of these 
problems together.

- Mike

On 3/21/14, 8:39 AM, Michael Wilde wrote:
> Mihael, All,
>
> I'd like to propose a Swift/K feature to provide a reasonable solution
> to this very common need for an app to return a dynamically determined
> set of files.
>
> file dynarry[ ] <runtime; prefix="myoutput.", suffix=".out", indexes="int">;
>
> dynarry = myApp(myArgs);
>
> The "runtime" mapper should initially have the same arguments and
> semantics (roughly) as simple_mapper, except for two new arguments:
>
>    "indexes" which determines how the matched file names will be indexed
> in the returned array
> "int" | "string" | "sequential"
> sequential: return the matched files as consecutive integer indices
> starting with 0
> int: expect the filename component between prefix and suffix to be
> convertible to an integer, and use that as the index
> eg myfile.012.out and myfile.204.out will return an array with the
> mapped files at indices 12 and 204.
> string: similar to int but return a string-indexed associative array.
> "sequential" is simplest and should be the default.
>
> "paths" which determines if the match names will be absolute or relative
> to the job dir
> paths="relative" | "absolute"
> (may not be needed if this can be determined uniquely based on the
> location argument.
>
> swiftwrap will allow array variables mapped in this manner to have any
> number of files, including zero. I.e. "runtime-mapped" files should not
> be listed in the expected output list for an app invocation. Its up to
> the users app to ensure that some files match the pattern. An additional
> arg could set e.g. minfiles and/or maxfiles, in which case the wrapper
> code needs to validate the count of files matched and returned, but not
> their exact names.
>
> We can call this mapper "experimental" until we validate its usability
> and suitability as a permanent feature. But as we hope to revise the
> entire mapper family and semantics, in a sense all mappers are subject
> to change.
>
> Mihael, is the definition sound, and how long would it take you to
> develop it?
>
> Thanks,
>
> - Mike
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago