[Swift-devel] discussion for swift output file path

[Ben Clifford]

The wrapper does not necessarily put all output files in one directory.

It puts them in a directory structure reflecting the submit-side directory 
structure. For example, if you map a file "a/b", it will end up in the 
shared directory as shared/a/b

This is exposed to the user, though, rather than being hidden as with the 
other uses of subdirectories.

There are a few places throughout the source code where its assumed that 
the path locally and remote path are basically the same (modulo base 
directory). I think its probably fairly straightforward to make it use 
different names, thoughLook for lines in vdl-int.k that look like this:

    task:transfer(srcprovider=provider, 
srchost=srchost, srcfile=filename, 
                                                srcdir=srcdir, 
desthost=host, destdir=destdir)
                                        )


You can put a different destdir in there, based on (for example) some hash 
of the filename. Simialrly hash the filenames when they are passed as 
inputs to wrapper.sh in the vdl:execute line in vdl-int.k

On Thu, 6 Nov 2008, Zhao Zhang wrote:

> Hi, All
> 
> I am working on integrate the Collective IO system and swift on BGP. Before
> that, for the purpose of put swift into production work,
> we need to change the output file path. For now, wrapper.sh would copy all
> output files to jobdir/shared/, on BGP, all output files will
> be written to one directory, which I am sure will cause the GPFS lock
> mechanism, thus introduce unacceptable latency.
> 
> So the easiest way to fix this is "make a hierarchical directory in shared/
> and we already did in info/ and jobs/". Several changes we need:
>     place-need-change
> diffculty
> 1. change vdl-int.k: create hierarchical directory in shared/,
> straightforward
> 2. change wrapper.sh: copy files from local ramdisck to GPFS using dd instead
> of cp,      straghtforward.
> 3. change somewhere in swift to make swift know where the data is, the path of
> the output file in jobdir/shared/    unknown
> 
> Any comments will be appreciated.
> 
> best wishes
> zhangzhao
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
>