[Swift-devel] direct mode

[Michael Wilde]

On 8/9/14, 3:45 PM, Mihael Hategan wrote:
> On Sat, 2014-08-09 at 13:54 -0500, Michael Wilde wrote:
>> On 8/7/14, 8:56 PM, Mihael Hategan wrote:
>>> ...
>>> It is fairly easy to implement a manual direct mode, where one would use
>>> direct:// in the mapper file specification. This requires a modification
>>> of filename()/filenames(), as well as skipping the staging of files with
>>> direct://.
>> Following up on this discussion from yesterday's meeting:
>>
>> On re-thinking this: I can see the value of the direct:// prefix,
>> especially if its prepended to the filename using a function, eg:
>>
>> file f <direct("/etc/group")>;
> It could just be
> file f <"direct:///etc/group">;
>
> It's in line with what is currently supported. For example:
>
> file f <"gsiftp://...">;
> or
> file f <"http://google.com/index.html">;
When the filename is in a var, e.g., fname, direct(fname) reads cleaner 
than strcat("direct://",fname).
But not by much, so perhaps set that idea aside.
>> This is preferable to the current practice of using strings instead of
>> files for direct access, as it preserves automatic dependency processing.
> I'm not sure what you mean. The dependency processing is independent of
> mapping and it happens at the variable level (i.e. the f variable).
Yes, but when passing string filenames, the return is not a filename, it 
is typically a dummy var of type external, and the outfile filename is 
passed as an input string. So the dependency processing doesnt help you 
when passing output filenames as strings.
>> This would be used when some files (e.g., large files that should be
>> moved to a site once per run) need to be handled differently than the
>> site's default data management strategy.
> Actually that is what happens in swift staging mode by default. Input
> files are only moved once to the site, and then a link is created in the
> job directory.
But not for provider staging, as far as I understand.
> In any event, my intent was to provide both a direct staging mode and a
> direct:// override in the URL scheme of individual files.
That sounds good.

- Mike

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago