[Swift-devel] Wrapper staging

[David Kelly]

I'd like to find a way to define different staging methods on a file-by-file basis. In some cases it would make sense to use dd to copy a file to the work directory. In other situations it may be better to create a symlink pointing to what you need. 


I'm thinking of something along these lines (names will probably get changed): 


file data <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">; // Will stage the data to the work directory 


file data2 <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">; // Will create a symlink 


Then based on the prefix you define, _swiftwrap.wrapperstaging can know exactly how to stage the file. 


Is it possible to create data provider aliases, making file-direct:// and file-indirect:// work exactly like file:// until it gets passed to the wrapper? 


Thanks, 
David 

----- Original Message -----


From: "Mihael Hategan" <hategan at mcs.anl.gov> 
To: "David Kelly" <davidk at ci.uchicago.edu> 
Cc: "Swift Devel" <swift-devel at ci.uchicago.edu> 
Sent: Wednesday, May 8, 2013 12:46:45 PM 
Subject: Re: Wrapper staging 

I'm a bit busy today and this week, so I might not get to fixing these 
until the weekend, but I'll try to answer some of your questions. 

On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote: 
> Hello, 
> 
> 
> Just a few random questions about wrapper staging: 
> 
> 
> How can I make gsiftp:// URIs get sent to _swiftwrap.wrapperstaging? I 
> have wrapper staging enabled, but Swift is trying to connect to the 
> server and retrieve the file, rather than just passing the URI to the 
> wrapper script. 

With wrapper staging enabled, swift should not try to *transfer* files, 
but mappers might connect to the server to figure out what files are 
there (i.e. to complete calls to Mapper.existing()). 

> 
> What does the property wrapper.staging.local.server do? 

It specifies what url prefix to pass to the wrapper for local files 
(i.e. accessible directly by swift). 

> 
> 
> From within _swiftwrap.wrapperstaging, how can I know the current 
> working directory of the user? I think I need this information to 
> handle relative path names. 

The wrapper should get a -urlprefix argument containing the CWD. The 
value of that arg is constructed as: ${wrapper.staging.local.server} + 
${CWD}. 

> 
> 
> I believe in trunk, filenames passed as arguments to Swift apps being 
> prepended with "__root__". What does this mean, and should swiftwrap 
> strip this out? 

No. There is a mapping from local file names to remote file names. It 
used to be that absolute files are mapped simply by removing the leading 
slash. So "/dir/file" on the swift side would become "dir/file" on the 
remote side. But then if there was a local "dir/file", it would also be 
"dir/file" on the remote side, causing a conflict. 

At some point in trunk, I changed that (and also the handling of "/../") 
in response to a bug that, I believe, Ketan submitted. The change 
involved mapping a leading slash to "__root__" and parent dir references 
to "__parent__". They have no special meaning as far as the wrapper or 
the application is concerned. They get the proper path through @filename 
and the likes, and swift stages the files in the places pointed by 
@filename. 

Mihael 




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130516/8160e9c9/attachment.html>