[Swift-devel] Wrapper staging

[Mihael Hategan]

This is beginning to sound awfully like CDM.

So I'm wondering if anybody remembers why we started this wrapper
staging business instead of using CDM.

To answer your question, yes. The wrapper staging code should just pass
the protocol to the wrapper, but mappers might have some trouble with
some things, so I might have to make some changes.

Mihael

On Thu, 2013-05-16 at 13:42 -0500, David Kelly wrote:
> I'd like to find a way to define different staging methods on a
> file-by-file basis. In some cases it would make sense to use dd to
> copy a file to the work directory. In other situations it may be
> better to create a symlink pointing to what you need. 
> 
> 
> I'm thinking of something along these lines (names will probably get changed): 
> 
> 
> file data <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">; // Will stage the data to the work directory 
> 
> 
> file data2 <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">; // Will create a symlink 
> 
> 
> Then based on the prefix you define, _swiftwrap.wrapperstaging can know exactly how to stage the file. 
> 
> 
> Is it possible to create data provider aliases, making file-direct:// and file-indirect:// work exactly like file:// until it gets passed to the wrapper? 
> 
> 
> Thanks, 
> David 
> 
> ----- Original Message -----
> 
> 
> From: "Mihael Hategan" <hategan at mcs.anl.gov> 
> To: "David Kelly" <davidk at ci.uchicago.edu> 
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu> 
> Sent: Wednesday, May 8, 2013 12:46:45 PM 
> Subject: Re: Wrapper staging 
> 
> I'm a bit busy today and this week, so I might not get to fixing these 
> until the weekend, but I'll try to answer some of your questions. 
> 
> On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote: 
> > Hello, 
> > 
> > 
> > Just a few random questions about wrapper staging: 
> > 
> > 
> > How can I make gsiftp:// URIs get sent to _swiftwrap.wrapperstaging? I 
> > have wrapper staging enabled, but Swift is trying to connect to the 
> > server and retrieve the file, rather than just passing the URI to the 
> > wrapper script. 
> 
> With wrapper staging enabled, swift should not try to *transfer* files, 
> but mappers might connect to the server to figure out what files are 
> there (i.e. to complete calls to Mapper.existing()). 
> 
> > 
> > What does the property wrapper.staging.local.server do? 
> 
> It specifies what url prefix to pass to the wrapper for local files 
> (i.e. accessible directly by swift). 
> 
> > 
> > 
> > From within _swiftwrap.wrapperstaging, how can I know the current 
> > working directory of the user? I think I need this information to 
> > handle relative path names. 
> 
> The wrapper should get a -urlprefix argument containing the CWD. The 
> value of that arg is constructed as: ${wrapper.staging.local.server} + 
> ${CWD}. 
> 
> > 
> > 
> > I believe in trunk, filenames passed as arguments to Swift apps being 
> > prepended with "__root__". What does this mean, and should swiftwrap 
> > strip this out? 
> 
> No. There is a mapping from local file names to remote file names. It 
> used to be that absolute files are mapped simply by removing the leading 
> slash. So "/dir/file" on the swift side would become "dir/file" on the 
> remote side. But then if there was a local "dir/file", it would also be 
> "dir/file" on the remote side, causing a conflict. 
> 
> At some point in trunk, I changed that (and also the handling of "/../") 
> in response to a bug that, I believe, Ketan submitted. The change 
> involved mapping a leading slash to "__root__" and parent dir references 
> to "__parent__". They have no special meaning as far as the wrapper or 
> the application is concerned. They get the proper path through @filename 
> and the likes, and swift stages the files in the places pointed by 
> @filename. 
> 
> Mihael 
> 
> 
> 
> 
> I'd like to find a way to define different staging methods on a
> file-by-file basis. In some cases it would make sense to use dd to
> copy a file to the work directory. In other situations it may be
> better to create a symlink pointing to what you need.
> 
> 
> I'm thinking of something along these lines (names will probably get
> changed):
> 
> 
> file data
> <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
>      // Will stage the data to the work directory
> file data2
> <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">; // Will create a symlink
> 
> 
> Then based on the prefix you define, _swiftwrap.wrapperstaging can
> know exactly how to stage the file. 
> 
> 
> Is it possible to create data provider aliases, making file-direct://
> and file-indirect:// work exactly like file:// until it gets passed to
> the wrapper?
> 
> 
> Thanks,
> David
> 
> 
> ______________________________________________________________________
>         From: "Mihael Hategan" <hategan at mcs.anl.gov>
>         To: "David Kelly" <davidk at ci.uchicago.edu>
>         Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
>         Sent: Wednesday, May 8, 2013 12:46:45 PM
>         Subject: Re: Wrapper staging
>         
>         I'm a bit busy today and this week, so I might not get to
>         fixing these
>         until the weekend, but I'll try to answer some of your
>         questions.
>         
>         On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
>         > Hello, 
>         > 
>         > 
>         > Just a few random questions about wrapper staging: 
>         > 
>         > 
>         > How can I make gsiftp:// URIs get sent to
>         _swiftwrap.wrapperstaging? I
>         > have wrapper staging enabled, but Swift is trying to connect
>         to the
>         > server and retrieve the file, rather than just passing the
>         URI to the
>         > wrapper script. 
>         
>         With wrapper staging enabled, swift should not try to
>         *transfer* files,
>         but mappers might connect to the server to figure out what
>         files are
>         there (i.e. to complete calls to Mapper.existing()).
>         
>         > 
>         > What does the property wrapper.staging.local.server do? 
>         
>         It specifies what url prefix to pass to the wrapper for local
>         files
>         (i.e. accessible directly by swift). 
>         
>         > 
>         > 
>         > From within _swiftwrap.wrapperstaging, how can I know the
>         current
>         > working directory of the user? I think I need this
>         information to
>         > handle relative path names. 
>         
>         The wrapper should get a -urlprefix argument containing the
>         CWD. The
>         value of that arg is constructed as:
>         ${wrapper.staging.local.server} +
>         ${CWD}.
>         
>         > 
>         > 
>         > I believe in trunk, filenames passed as arguments to Swift
>         apps being
>         > prepended with "__root__". What does this mean, and should
>         swiftwrap
>         > strip this out? 
>         
>         No. There is a mapping from local file names to remote file
>         names. It
>         used to be that absolute files are mapped simply by removing
>         the leading
>         slash. So "/dir/file" on the swift side would become
>         "dir/file" on the
>         remote side. But then if there was a local "dir/file", it
>         would also be
>         "dir/file" on the remote side, causing a conflict.
>         
>         At some point in trunk, I changed that (and also the handling
>         of "/../")
>         in response to a bug that, I believe, Ketan submitted. The
>         change
>         involved mapping a leading slash to "__root__" and parent dir
>         references
>         to "__parent__". They have no special meaning as far as the
>         wrapper or
>         the application is concerned. They get the proper path through
>         @filename
>         and the likes, and swift stages the files in the places
>         pointed by
>         @filename.
>         
>         Mihael
>         
> 
>