[Swift-devel] Multiple output files

[Mihael Hategan]

Almost there.

It works for non-provider staging.
I'm working on provider staging now. The difficulty is in that providers
must now support glob-pattern staging with a twist. The twist is that
local and remote path names are different. For example, without glob
patterns, a stageout could look like this:

__root__/dir/a.txt -> /dir/a.txt

With globs, something like this is possible:

__root__/dir/a_????_b_????.txt -> /dir/a_????_b_????.txt

In this case the code needs to recursively glob things and substitute
each glob group in the destination for the respective matching glob in
the source.

Luckily there are only two providers that support staging at this point:
local and coasters. Unfortunately this has to be implemented twice: once
in Java for local, and once in Perl for coasters.

Mihael

On Mon, 2014-03-24 at 08:46 -0700, Mihael Hategan wrote:
> An update on this.
> 
> I'm still working on it, but here is the basic idea:
> 
> - non-static mappers now support a method that, given a data type,
> returns a list of glob patterns that can be used to search for files
> that could be mapped by that mapper. The list (as opposed to one glob
> pattern) is necessary because there might be cases when you have:
>     type s { file a; file[] b};
>     s[] x;
>     Then s[] could match either s_????.a or s_????.????
> - this (possibly empty) list gets sent to _swiftwrap
> - after the job is done, _swiftwrap creates a list of files matching
> those patterns
> - swift-int copies that list back and the files in it and uses the list
> to populate data in a fashion similar to what is done for input
> variables
> 
> This is without provider staging.
> 
> For provider staging, providers that support staging need to be modified
> to support staging out of files using glob patterns. There might be some
> complications there due to the local vs. remote path naming conventions.
> 
> Mihael 
> 
> On Sat, 2014-03-22 at 14:08 -0500, Michael Wilde wrote:
> > On 3/22/14, 1:38 PM, Mihael Hategan wrote:
> > > My opinion is that this problem is NOT a language/mapper issue. It is an
> > > issue of implementation: how do you get the information about files to a
> > > place where it can be used.
> > >
> > > So I believe that whether we add a new mapper or make it work with
> > > existing mappers, we still need to fix that other more complex problem.
> > > This is the reason why I believe we shouldn't add anything to the
> > > language.
> > Mihael and I discussed this in a chat just now, and I think we are in 
> > fact *in* sync.
> > So he's going to push forward on this.
> > 
> > - Mike
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel