[Swift-devel] Wrapper staging

[Michael Wilde]

Doesn't "file://" already map to the "local" data provider? If so, can you simply replicate that entry in the provider alias list to create additional aliases to "local" as you suggest below, David?

- Mike

----- Original Message -----
> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: "Mihael Hategan" <hategan at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Thursday, May 16, 2013 1:42:41 PM
> Subject: Re: [Swift-devel] Wrapper staging
> 
> 
> 
> I'd like to find a way to define different staging methods on a
> file-by-file basis. In some cases it would make sense to use dd to
> copy a file to the work directory. In other situations it may be
> better to create a symlink pointing to what you need.
> 
> 
> I'm thinking of something along these lines (names will probably get
> changed):
> 
> 
> file data
> <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> // Will stage the data to the work directory
> 
> 
> file data2
> <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> // Will create a symlink
> 
> 
> Then based on the prefix you define, _swiftwrap.wrapperstaging can
> know exactly how to stage the file.
> 
> 
> Is it possible to create data provider aliases, making file-direct://
> and file-indirect:// work exactly like file:// until it gets passed
> to the wrapper?
> 
> 
> Thanks,
> David
> 
> 
> 
> 
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Wednesday, May 8, 2013 12:46:45 PM
> Subject: Re: Wrapper staging
> 
> I'm a bit busy today and this week, so I might not get to fixing
> these
> until the weekend, but I'll try to answer some of your questions.
> 
> On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> > Hello,
> > 
> > 
> > Just a few random questions about wrapper staging:
> > 
> > 
> > How can I make gsiftp:// URIs get sent to
> > _swiftwrap.wrapperstaging? I
> > have wrapper staging enabled, but Swift is trying to connect to the
> > server and retrieve the file, rather than just passing the URI to
> > the
> > wrapper script.
> 
> With wrapper staging enabled, swift should not try to *transfer*
> files,
> but mappers might connect to the server to figure out what files are
> there (i.e. to complete calls to Mapper.existing()).
> 
> > 
> > What does the property wrapper.staging.local.server do?
> 
> It specifies what url prefix to pass to the wrapper for local files
> (i.e. accessible directly by swift).
> 
> > 
> > 
> > From within _swiftwrap.wrapperstaging, how can I know the current
> > working directory of the user? I think I need this information to
> > handle relative path names.
> 
> The wrapper should get a -urlprefix argument containing the CWD. The
> value of that arg is constructed as: ${wrapper.staging.local.server}
> +
> ${CWD}.
> 
> > 
> > 
> > I believe in trunk, filenames passed as arguments to Swift apps
> > being
> > prepended with "__root__". What does this mean, and should
> > swiftwrap
> > strip this out?
> 
> No. There is a mapping from local file names to remote file names. It
> used to be that absolute files are mapped simply by removing the
> leading
> slash. So "/dir/file" on the swift side would become "dir/file" on
> the
> remote side. But then if there was a local "dir/file", it would also
> be
> "dir/file" on the remote side, causing a conflict.
> 
> At some point in trunk, I changed that (and also the handling of
> "/../")
> in response to a bug that, I believe, Ketan submitted. The change
> involved mapping a leading slash to "__root__" and parent dir
> references
> to "__parent__". They have no special meaning as far as the wrapper
> or
> the application is concerned. They get the proper path through
> @filename
> and the likes, and swift stages the files in the places pointed by
> @filename.
> 
> Mihael
> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>