[Swift-devel] Multiple output files

[Michael Wilde]

Excellent!

The path-difference challenge sounds like the same things we faced in 
CDM, revisited.

I suspect we can find ways to smooth this out in a way that works 
naturally for users.

- Mike


On 3/26/14, 1:24 PM, Mihael Hategan wrote:
> Almost there.
>
> It works for non-provider staging.
> I'm working on provider staging now. The difficulty is in that providers
> must now support glob-pattern staging with a twist. The twist is that
> local and remote path names are different. For example, without glob
> patterns, a stageout could look like this:
>
> __root__/dir/a.txt -> /dir/a.txt
>
> With globs, something like this is possible:
>
> __root__/dir/a_????_b_????.txt -> /dir/a_????_b_????.txt
>
> In this case the code needs to recursively glob things and substitute
> each glob group in the destination for the respective matching glob in
> the source.
>
> Luckily there are only two providers that support staging at this point:
> local and coasters. Unfortunately this has to be implemented twice: once
> in Java for local, and once in Perl for coasters.
>
> Mihael
>
> On Mon, 2014-03-24 at 08:46 -0700, Mihael Hategan wrote:
>> An update on this.
>>
>> I'm still working on it, but here is the basic idea:
>>
>> - non-static mappers now support a method that, given a data type,
>> returns a list of glob patterns that can be used to search for files
>> that could be mapped by that mapper. The list (as opposed to one glob
>> pattern) is necessary because there might be cases when you have:
>>      type s { file a; file[] b};
>>      s[] x;
>>      Then s[] could match either s_????.a or s_????.????
>> - this (possibly empty) list gets sent to _swiftwrap
>> - after the job is done, _swiftwrap creates a list of files matching
>> those patterns
>> - swift-int copies that list back and the files in it and uses the list
>> to populate data in a fashion similar to what is done for input
>> variables
>>
>> This is without provider staging.
>>
>> For provider staging, providers that support staging need to be modified
>> to support staging out of files using glob patterns. There might be some
>> complications there due to the local vs. remote path naming conventions.
>>
>> Mihael
>>
>> On Sat, 2014-03-22 at 14:08 -0500, Michael Wilde wrote:
>>> On 3/22/14, 1:38 PM, Mihael Hategan wrote:
>>>> My opinion is that this problem is NOT a language/mapper issue. It is an
>>>> issue of implementation: how do you get the information about files to a
>>>> place where it can be used.
>>>>
>>>> So I believe that whether we add a new mapper or make it work with
>>>> existing mappers, we still need to fix that other more complex problem.
>>>> This is the reason why I believe we shouldn't add anything to the
>>>> language.
>>> Mihael and I discussed this in a chat just now, and I think we are in
>>> fact *in* sync.
>>> So he's going to push forward on this.
>>>
>>> - Mike
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago