[Swift-devel] Multiple output files

[Michael Wilde]

Hi Jonathan,

On 3/20/14, 6:04 PM, Jonathan Ozik wrote:
> Mike,
>
> Thank you for the detailed information.
> Regarding the "collect files of this pattern into an array" semantics, is the "file system mapper" not intended for this?
No, its not. Mihael and others may need to correct me here, but 
basically the issue is this:

An app *can* return multiple files - even an array of files - but not an 
array of files whose names and count is not known before the app is 
launched.

The User Guide does not yet cover this adequately, but output mappings 
may be static or dynamic. Static means the names and quantify of files 
is determined when the mapping is made. Dynamic means the mapping is 
made on demand, as files are created. Such mappings for example can be 
used to map the elements of the array when they are filled within a 
foreach loop by an app that returns one file per invocation.

What we do not have - but have long known that we need - is the ability 
to declare that all the files created by a *single* app invocation which 
match a specified pattern be returned as an array.

Mihael: is this something you could implement in the near future - after 
we agree on the semantics?

Justin, Tim, do you want to comment on this from a Swift/T perspective?

Thanks,

- Mike
> I think I'm stumbling on the notion of what output, in terms of files, can be produced using an "app" element in a swift script. Most of the examples have stdout (or stderr) pointing to a file, but I'm not sure I've found an example where side effects (e.g., files produced by a process) can be retrieved. Is this possible?
>
> Jonathan
>
> On Mar 19, 2014, at 7:03 PM, Michael Wilde <wilde at anl.gov> wrote:
>
>> Hi Jonathan,
>>
>> You are thinking about this exactly right, and have (quickly) hit one of our programmability weaknesses.
>>
>> At the moment, there is no good way to do this. We have had much discussion on it, though, and plan to create such "collect files of this pattern into an array" semantics.
>>
>> What we do for now is one of these two work-arounds:
>>
>> - write a single tarfile as output (in a wrapper script for the app, which tars files of the appropriate pattern, like "*.simout"
>>
>> - write the files directly to a specific (shared) directory instead of the Swift "app task directory" (called "job directory" in the current User Guide.  Then return a single file with a list of these file names, and map that using an array_mapper if the results need to be passed to a next stage
>>
>> We should see if we can get a prototype of such a feature to you in short order. But hopefully just to get things working, one of the above methods will suffice for you, for now.
>>
>> Im cc'ing swift-devel to see what we can do.
>>
>> Thanks for reminding us of this fairly common need!
>>
>> - Mike
>>
>> On 3/19/14, 5:57 PM, Jonathan Ozik wrote:
>>> Mike,
>>>
>>> Perhaps I'm not thinking about this correctly, but I'm trying to figure out how to collect output files that are generated by a simulation run. The scenario is that the Repast executable is run and, after it is run, there are files that are output into some location. There can be an arbitrary number of such output files per simulation run but, if necessary, it would be possible to pre-specify which files to look for. Is there a simple way to indicate to swift that it should collect all the files matching a particular pattern within a directory? It looks like the mappers might work here but as far as I understand the mappers are defined prior to calls to executables. Again, I might just not be thinking about this in a "swift" enough way.
>>>
>>> Jonathan
>>>
>> -- 
>> Michael Wilde
>> Mathematics and Computer Science          Computation Institute
>> Argonne National Laboratory               The University of Chicago
>>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago