[Swift-devel] Wrapper staging

[David Kelly]

When I try to use file-direct:// with no modifications I get this: 

Progress: time: Thu, 16 May 2013 19:04:23 +0000 
Execution failed: 
java.lang.RuntimeException: java.lang.RuntimeException: Could not instantiate file resource 
Caused by: 
No 'file-direct' provider or alias found. Available providers: [cobalt, ssh-cl, gsiftp, coaster, webdav, slurm, dcache, ssh, gt2, condor, http, coaster-persistent, pbs, ftp, lsf, gsiftp-old, local, sge]. Aliases: condor <-> condorlocal; cobalt <-> cobaltlocal; gsiftp <-> gridftp; pbs <-> pbslocal; gsiftp-old <-> gridftp-old; local <-> file; 

So I see in that message that local <-> file are aliases. I wasn't sure exactly where those were defined, but I think I found it in modules/provider-local/resources/cog-provider.properties. I added this line: 

alias=file-direct:local 

Then I rebuilt, and it seems to be getting to the wrapper now. Thanks. 

David 
----- Original Message -----

> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Mihael Hategan"
> <hategan at mcs.anl.gov>
> Sent: Thursday, May 16, 2013 2:00:54 PM
> Subject: Re: [Swift-devel] Wrapper staging

> Doesn't "file://" already map to the "local" data provider? If so,
> can you simply replicate that entry in the provider alias list to
> create additional aliases to "local" as you suggest below, David?

> - Mike

> ----- Original Message -----
> > From: "David Kelly" <davidk at ci.uchicago.edu>
> > To: "Mihael Hategan" <hategan at mcs.anl.gov>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Thursday, May 16, 2013 1:42:41 PM
> > Subject: Re: [Swift-devel] Wrapper staging
> >
> >
> >
> > I'd like to find a way to define different staging methods on a
> > file-by-file basis. In some cases it would make sense to use dd to
> > copy a file to the work directory. In other situations it may be
> > better to create a symlink pointing to what you need.
> >
> >
> > I'm thinking of something along these lines (names will probably
> > get
> > changed):
> >
> >
> > file data
> > <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> > // Will stage the data to the work directory
> >
> >
> > file data2
> > <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> > // Will create a symlink
> >
> >
> > Then based on the prefix you define, _swiftwrap.wrapperstaging can
> > know exactly how to stage the file.
> >
> >
> > Is it possible to create data provider aliases, making
> > file-direct://
> > and file-indirect:// work exactly like file:// until it gets passed
> > to the wrapper?
> >
> >
> > Thanks,
> > David
> >
> >
> >
> >
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Wednesday, May 8, 2013 12:46:45 PM
> > Subject: Re: Wrapper staging
> >
> > I'm a bit busy today and this week, so I might not get to fixing
> > these
> > until the weekend, but I'll try to answer some of your questions.
> >
> > On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> > > Hello,
> > >
> > >
> > > Just a few random questions about wrapper staging:
> > >
> > >
> > > How can I make gsiftp:// URIs get sent to
> > > _swiftwrap.wrapperstaging? I
> > > have wrapper staging enabled, but Swift is trying to connect to
> > > the
> > > server and retrieve the file, rather than just passing the URI to
> > > the
> > > wrapper script.
> >
> > With wrapper staging enabled, swift should not try to *transfer*
> > files,
> > but mappers might connect to the server to figure out what files
> > are
> > there (i.e. to complete calls to Mapper.existing()).
> >
> > >
> > > What does the property wrapper.staging.local.server do?
> >
> > It specifies what url prefix to pass to the wrapper for local files
> > (i.e. accessible directly by swift).
> >
> > >
> > >
> > > From within _swiftwrap.wrapperstaging, how can I know the current
> > > working directory of the user? I think I need this information to
> > > handle relative path names.
> >
> > The wrapper should get a -urlprefix argument containing the CWD.
> > The
> > value of that arg is constructed as:
> > ${wrapper.staging.local.server}
> > +
> > ${CWD}.
> >
> > >
> > >
> > > I believe in trunk, filenames passed as arguments to Swift apps
> > > being
> > > prepended with "__root__". What does this mean, and should
> > > swiftwrap
> > > strip this out?
> >
> > No. There is a mapping from local file names to remote file names.
> > It
> > used to be that absolute files are mapped simply by removing the
> > leading
> > slash. So "/dir/file" on the swift side would become "dir/file" on
> > the
> > remote side. But then if there was a local "dir/file", it would
> > also
> > be
> > "dir/file" on the remote side, causing a conflict.
> >
> > At some point in trunk, I changed that (and also the handling of
> > "/../")
> > in response to a bug that, I believe, Ketan submitted. The change
> > involved mapping a leading slash to "__root__" and parent dir
> > references
> > to "__parent__". They have no special meaning as far as the wrapper
> > or
> > the application is concerned. They get the proper path through
> > @filename
> > and the likes, and swift stages the files in the places pointed by
> > @filename.
> >
> > Mihael
> >
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130516/530b7cf5/attachment.html>