[Swift-devel] Wrapper staging

[Michael Wilde]

We started wrapper staging for the broader goal of revisiting and improving the way we do all kinds of data management.

An initial goal is to allow a combination of direct access and local copies based on much simpler defaults, and the ability to specify a simple default in swift.properties and override at the point of mapping.

Later goals include using Globus staging tools (globus-url-copy and Globus Online) instead of the outdated GridFTP provider based on JGlobus.

Further goals include using catalog-look for files and doing compute node file caching, and improving on the current CDM implementation.

We want to create data management modes that will work in both Swift1 (/K) and Swift2 (/T) and which are cleanly isolated from the core language implementation.

- Mike

----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Thursday, May 16, 2013 1:53:07 PM
> Subject: Re: [Swift-devel] Wrapper staging
> 
> This is beginning to sound awfully like CDM.
> 
> So I'm wondering if anybody remembers why we started this wrapper
> staging business instead of using CDM.
> 
> To answer your question, yes. The wrapper staging code should just
> pass
> the protocol to the wrapper, but mappers might have some trouble with
> some things, so I might have to make some changes.
> 
> Mihael
> 
> On Thu, 2013-05-16 at 13:42 -0500, David Kelly wrote:
> > I'd like to find a way to define different staging methods on a
> > file-by-file basis. In some cases it would make sense to use dd to
> > copy a file to the work directory. In other situations it may be
> > better to create a symlink pointing to what you need.
> > 
> > 
> > I'm thinking of something along these lines (names will probably
> > get changed):
> > 
> > 
> > file data
> > <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> > // Will stage the data to the work directory
> > 
> > 
> > file data2
> > <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> > // Will create a symlink
> > 
> > 
> > Then based on the prefix you define, _swiftwrap.wrapperstaging can
> > know exactly how to stage the file.
> > 
> > 
> > Is it possible to create data provider aliases, making
> > file-direct:// and file-indirect:// work exactly like file://
> > until it gets passed to the wrapper?
> > 
> > 
> > Thanks,
> > David
> > 
> > ----- Original Message -----
> > 
> > 
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Wednesday, May 8, 2013 12:46:45 PM
> > Subject: Re: Wrapper staging
> > 
> > I'm a bit busy today and this week, so I might not get to fixing
> > these
> > until the weekend, but I'll try to answer some of your questions.
> > 
> > On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> > > Hello,
> > > 
> > > 
> > > Just a few random questions about wrapper staging:
> > > 
> > > 
> > > How can I make gsiftp:// URIs get sent to
> > > _swiftwrap.wrapperstaging? I
> > > have wrapper staging enabled, but Swift is trying to connect to
> > > the
> > > server and retrieve the file, rather than just passing the URI to
> > > the
> > > wrapper script.
> > 
> > With wrapper staging enabled, swift should not try to *transfer*
> > files,
> > but mappers might connect to the server to figure out what files
> > are
> > there (i.e. to complete calls to Mapper.existing()).
> > 
> > > 
> > > What does the property wrapper.staging.local.server do?
> > 
> > It specifies what url prefix to pass to the wrapper for local files
> > (i.e. accessible directly by swift).
> > 
> > > 
> > > 
> > > From within _swiftwrap.wrapperstaging, how can I know the current
> > > working directory of the user? I think I need this information to
> > > handle relative path names.
> > 
> > The wrapper should get a -urlprefix argument containing the CWD.
> > The
> > value of that arg is constructed as:
> > ${wrapper.staging.local.server} +
> > ${CWD}.
> > 
> > > 
> > > 
> > > I believe in trunk, filenames passed as arguments to Swift apps
> > > being
> > > prepended with "__root__". What does this mean, and should
> > > swiftwrap
> > > strip this out?
> > 
> > No. There is a mapping from local file names to remote file names.
> > It
> > used to be that absolute files are mapped simply by removing the
> > leading
> > slash. So "/dir/file" on the swift side would become "dir/file" on
> > the
> > remote side. But then if there was a local "dir/file", it would
> > also be
> > "dir/file" on the remote side, causing a conflict.
> > 
> > At some point in trunk, I changed that (and also the handling of
> > "/../")
> > in response to a bug that, I believe, Ketan submitted. The change
> > involved mapping a leading slash to "__root__" and parent dir
> > references
> > to "__parent__". They have no special meaning as far as the wrapper
> > or
> > the application is concerned. They get the proper path through
> > @filename
> > and the likes, and swift stages the files in the places pointed by
> > @filename.
> > 
> > Mihael
> > 
> > 
> > 
> > 
> > I'd like to find a way to define different staging methods on a
> > file-by-file basis. In some cases it would make sense to use dd to
> > copy a file to the work directory. In other situations it may be
> > better to create a symlink pointing to what you need.
> > 
> > 
> > I'm thinking of something along these lines (names will probably
> > get
> > changed):
> > 
> > 
> > file data
> > <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> >      // Will stage the data to the work directory
> > file data2
> > <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> > // Will create a symlink
> > 
> > 
> > Then based on the prefix you define, _swiftwrap.wrapperstaging can
> > know exactly how to stage the file.
> > 
> > 
> > Is it possible to create data provider aliases, making
> > file-direct://
> > and file-indirect:// work exactly like file:// until it gets passed
> > to
> > the wrapper?
> > 
> > 
> > Thanks,
> > David
> > 
> > 
> > ______________________________________________________________________
> >         From: "Mihael Hategan" <hategan at mcs.anl.gov>
> >         To: "David Kelly" <davidk at ci.uchicago.edu>
> >         Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> >         Sent: Wednesday, May 8, 2013 12:46:45 PM
> >         Subject: Re: Wrapper staging
> >         
> >         I'm a bit busy today and this week, so I might not get to
> >         fixing these
> >         until the weekend, but I'll try to answer some of your
> >         questions.
> >         
> >         On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> >         > Hello,
> >         > 
> >         > 
> >         > Just a few random questions about wrapper staging:
> >         > 
> >         > 
> >         > How can I make gsiftp:// URIs get sent to
> >         _swiftwrap.wrapperstaging? I
> >         > have wrapper staging enabled, but Swift is trying to
> >         > connect
> >         to the
> >         > server and retrieve the file, rather than just passing
> >         > the
> >         URI to the
> >         > wrapper script.
> >         
> >         With wrapper staging enabled, swift should not try to
> >         *transfer* files,
> >         but mappers might connect to the server to figure out what
> >         files are
> >         there (i.e. to complete calls to Mapper.existing()).
> >         
> >         > 
> >         > What does the property wrapper.staging.local.server do?
> >         
> >         It specifies what url prefix to pass to the wrapper for
> >         local
> >         files
> >         (i.e. accessible directly by swift).
> >         
> >         > 
> >         > 
> >         > From within _swiftwrap.wrapperstaging, how can I know the
> >         current
> >         > working directory of the user? I think I need this
> >         information to
> >         > handle relative path names.
> >         
> >         The wrapper should get a -urlprefix argument containing the
> >         CWD. The
> >         value of that arg is constructed as:
> >         ${wrapper.staging.local.server} +
> >         ${CWD}.
> >         
> >         > 
> >         > 
> >         > I believe in trunk, filenames passed as arguments to
> >         > Swift
> >         apps being
> >         > prepended with "__root__". What does this mean, and
> >         > should
> >         swiftwrap
> >         > strip this out?
> >         
> >         No. There is a mapping from local file names to remote file
> >         names. It
> >         used to be that absolute files are mapped simply by
> >         removing
> >         the leading
> >         slash. So "/dir/file" on the swift side would become
> >         "dir/file" on the
> >         remote side. But then if there was a local "dir/file", it
> >         would also be
> >         "dir/file" on the remote side, causing a conflict.
> >         
> >         At some point in trunk, I changed that (and also the
> >         handling
> >         of "/../")
> >         in response to a bug that, I believe, Ketan submitted. The
> >         change
> >         involved mapping a leading slash to "__root__" and parent
> >         dir
> >         references
> >         to "__parent__". They have no special meaning as far as the
> >         wrapper or
> >         the application is concerned. They get the proper path
> >         through
> >         @filename
> >         and the likes, and swift stages the files in the places
> >         pointed by
> >         @filename.
> >         
> >         Mihael
> >         
> > 
> > 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>