[Swift-devel] Wrapper staging

[Michael Wilde]

Cool.  It might be interesting to see if the "//" is wired-in or optional. If it can be omitted, then you might be able to make the specification more compact and less noisy:

  file data<"direct:/home/davidkelly999/staging/dd-absolute/data.txt">;
  file data<"direct:data.txt">;
  file data<"stage:data.txt">;

- Mike

----- Original Message -----
> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Mihael Hategan" <hategan at mcs.anl.gov>
> Sent: Thursday, May 16, 2013 2:19:02 PM
> Subject: Re: [Swift-devel] Wrapper staging
> 
> 
> When I try to use file-direct:// with no modifications I get this:
> 
> 
> 
> Progress: time: Thu, 16 May 2013 19:04:23 +0000
> Execution failed:
> java.lang.RuntimeException: java.lang.RuntimeException: Could not
> instantiate file resource
> Caused by:
> No 'file-direct' provider or alias found. Available providers:
> [cobalt, ssh-cl, gsiftp, coaster, webdav, slurm, dcache, ssh, gt2,
> condor, http, coaster-persistent, pbs, ftp, lsf, gsiftp-old, local,
> sge]. Aliases: condor <-> condorlocal; cobalt <-> cobaltlocal;
> gsiftp <-> gridftp; pbs <-> pbslocal; gsiftp-old <-> gridftp-old;
> local <-> file;
> 
> 
> So I see in that message that local <-> file are aliases. I wasn't
> sure exactly where those were defined, but I think I found it in
> modules/provider-local/resources/cog-provider.properties. I added
> this line:
> 
> 
> alias=file-direct:local
> 
> 
> Then I rebuilt, and it seems to be getting to the wrapper now.
> Thanks.
> 
> 
> David
> ----- Original Message -----
> 
> 
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Mihael Hategan"
> <hategan at mcs.anl.gov>
> Sent: Thursday, May 16, 2013 2:00:54 PM
> Subject: Re: [Swift-devel] Wrapper staging
> 
> Doesn't "file://" already map to the "local" data provider? If so,
> can you simply replicate that entry in the provider alias list to
> create additional aliases to "local" as you suggest below, David?
> 
> - Mike
> 
> ----- Original Message -----
> > From: "David Kelly" <davidk at ci.uchicago.edu>
> > To: "Mihael Hategan" <hategan at mcs.anl.gov>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Thursday, May 16, 2013 1:42:41 PM
> > Subject: Re: [Swift-devel] Wrapper staging
> > 
> > 
> > 
> > I'd like to find a way to define different staging methods on a
> > file-by-file basis. In some cases it would make sense to use dd to
> > copy a file to the work directory. In other situations it may be
> > better to create a symlink pointing to what you need.
> > 
> > 
> > I'm thinking of something along these lines (names will probably
> > get
> > changed):
> > 
> > 
> > file data
> > <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> > // Will stage the data to the work directory
> > 
> > 
> > file data2
> > <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> > // Will create a symlink
> > 
> > 
> > Then based on the prefix you define, _swiftwrap.wrapperstaging can
> > know exactly how to stage the file.
> > 
> > 
> > Is it possible to create data provider aliases, making
> > file-direct://
> > and file-indirect:// work exactly like file:// until it gets passed
> > to the wrapper?
> > 
> > 
> > Thanks,
> > David
> > 
> > 
> > 
> > 
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Wednesday, May 8, 2013 12:46:45 PM
> > Subject: Re: Wrapper staging
> > 
> > I'm a bit busy today and this week, so I might not get to fixing
> > these
> > until the weekend, but I'll try to answer some of your questions.
> > 
> > On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> > > Hello,
> > > 
> > > 
> > > Just a few random questions about wrapper staging:
> > > 
> > > 
> > > How can I make gsiftp:// URIs get sent to
> > > _swiftwrap.wrapperstaging? I
> > > have wrapper staging enabled, but Swift is trying to connect to
> > > the
> > > server and retrieve the file, rather than just passing the URI to
> > > the
> > > wrapper script.
> > 
> > With wrapper staging enabled, swift should not try to *transfer*
> > files,
> > but mappers might connect to the server to figure out what files
> > are
> > there (i.e. to complete calls to Mapper.existing()).
> > 
> > > 
> > > What does the property wrapper.staging.local.server do?
> > 
> > It specifies what url prefix to pass to the wrapper for local files
> > (i.e. accessible directly by swift).
> > 
> > > 
> > > 
> > > From within _swiftwrap.wrapperstaging, how can I know the current
> > > working directory of the user? I think I need this information to
> > > handle relative path names.
> > 
> > The wrapper should get a -urlprefix argument containing the CWD.
> > The
> > value of that arg is constructed as:
> > ${wrapper.staging.local.server}
> > +
> > ${CWD}.
> > 
> > > 
> > > 
> > > I believe in trunk, filenames passed as arguments to Swift apps
> > > being
> > > prepended with "__root__". What does this mean, and should
> > > swiftwrap
> > > strip this out?
> > 
> > No. There is a mapping from local file names to remote file names.
> > It
> > used to be that absolute files are mapped simply by removing the
> > leading
> > slash. So "/dir/file" on the swift side would become "dir/file" on
> > the
> > remote side. But then if there was a local "dir/file", it would
> > also
> > be
> > "dir/file" on the remote side, causing a conflict.
> > 
> > At some point in trunk, I changed that (and also the handling of
> > "/../")
> > in response to a bug that, I believe, Ketan submitted. The change
> > involved mapping a leading slash to "__root__" and parent dir
> > references
> > to "__parent__". They have no special meaning as far as the wrapper
> > or
> > the application is concerned. They get the proper path through
> > @filename
> > and the likes, and swift stages the files in the places pointed by
> > @filename.
> > 
> > Mihael
> > 
> > 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> 
>