[Swift-devel] direct mode

[Mihael Hategan]

On Sat, 2014-08-09 at 13:54 -0500, Michael Wilde wrote:
> On 8/7/14, 8:56 PM, Mihael Hategan wrote:
> > ...
> > It is fairly easy to implement a manual direct mode, where one would use
> > direct:// in the mapper file specification. This requires a modification
> > of filename()/filenames(), as well as skipping the staging of files with
> > direct://.
> Following up on this discussion from yesterday's meeting:
> 
> On re-thinking this: I can see the value of the direct:// prefix, 
> especially if its prepended to the filename using a function, eg:
> 
> file f <direct("/etc/group")>;

It could just be 
file f <"direct:///etc/group">;

It's in line with what is currently supported. For example:

file f <"gsiftp://...">;
or
file f <"http://google.com/index.html">;

> 
> This is preferable to the current practice of using strings instead of 
> files for direct access, as it preserves automatic dependency processing.

I'm not sure what you mean. The dependency processing is independent of
mapping and it happens at the variable level (i.e. the f variable).

> 
> This would be used when some files (e.g., large files that should be 
> moved to a site once per run) need to be handled differently than the 
> site's default data management strategy.

Actually that is what happens in swift staging mode by default. Input
files are only moved once to the site, and then a link is created in the
job directory.

> 
> This could generalize by giving each data management stratetegy a name; 
> each site would have a default strategy declared in the site entry, and 
> each file could get a different strategy by specifying a prefix, either 
> in the filename URL protocol field or in the URL via a function like 
> "direct( )", above.

To a large extend this is the current behavior. Sites have a staging
mode, and that mode refers to how files on the swift client side are
staged. When a path is specified in the mapper, it defaults to referring
to a client-local file. This can be overridden using a full URL (e.g.
gsiftp://site.com/...).

> 
> Thoughts?

We might need to refine this further. I'm not sure I fully understand
what you are saying.

In any event, my intent was to provide both a direct staging mode and a
direct:// override in the URL scheme of individual files. 

Mihael