[Swift-devel] direct mode

[Mihael Hategan]

On Sat, 2014-08-09 at 15:53 -0500, Michael Wilde wrote:
> On 8/9/14, 3:45 PM, Mihael Hategan wrote:
> >>
> >> file f <direct("/etc/group")>;
> > It could just be
> > file f <"direct:///etc/group">;
> >
> > It's in line with what is currently supported. For example:
> >
> > file f <"gsiftp://...">;
> > or
> > file f <"http://google.com/index.html">;
> When the filename is in a var, e.g., fname, direct(fname) reads cleaner 
> than strcat("direct://",fname).

I see. Yeah, but you can:
- have "direct://" in fname to begin with
- say "direct://" + fname. We've had the concatenation operator for a
while.

> But not by much, so perhaps set that idea aside.
> >> This is preferable to the current practice of using strings instead of
> >> files for direct access, as it preserves automatic dependency processing.
> > I'm not sure what you mean. The dependency processing is independent of
> > mapping and it happens at the variable level (i.e. the f variable).

> Yes, but when passing string filenames, the return is not a filename, it 
> is typically a dummy var of type external, and the outfile filename is 
> passed as an input string. So the dependency processing doesnt help you 
> when passing output filenames as strings.

Well, right. Though I suspect that passing string file names was a hack
intended to emulate what direct:// would do. So, yes, this would be a
proper implementation of direct mode with dependency tracking.

> >> This would be used when some files (e.g., large files that should be
> >> moved to a site once per run) need to be handled differently than the
> >> site's default data management strategy.

> > Actually that is what happens in swift staging mode by default. Input
> > files are only moved once to the site, and then a link is created in the
> > job directory.

> But not for provider staging, as far as I understand.

Right. The lack of caching in provider staging is, I believe, a separate
issue that needs to be solved by adding caching to provider staging
(well, coasters). 

But the issue of optimizing staging in general is something that we
probably need to discuss further. My fear is that if we move the
discussion in that direction, we might not get to a direct mode very
soon.

Mihael