[Swift-devel] remote file/directory stuff (bug 22)

[Ben Clifford]

I want to work out what the 0.2-scoped work for this should be.

The description in bugzilla for bug 22 is a little misleading in as much 
as I want this to solve a specific app problem that Nika is having, and 
I've changed my understanding of what her problem is since I submitted 
that bug.

There's a broad issue of how to deal with applications that want a 
particular remote directory/filename layout for input and output; in the 
shorter term, I'm using Nika's app as the motivation for 0.2.


This is the general problem that Nika has, as far as I see it:

One of the programs insists on placing output files in the same directory 
as the input, with the name being the same as the input file name, with an 
interface that looks something like this:

 # the input files in a directory
 $ ls foodir/
 m002_am1

 # run the job
 $ antch foodir/m002_am1

 # now the directory that had the input file has a bunch of 
 # intermediate files, as well as the output files
 $ ls foodir/
 m002_am1
 m002_am1.crd
 m002_am1.prm
 m002_am1.rtf
 <some working files>

Swift cannot tell this program where to place the output files on the 
remote end. However, given knowledge about where swift has staged the 
input file into the job directory, it is straightforward to specify where 
the interesting output files have appeared under the job directory.

At present the syntax in swift is this: (actually I've cut out other 
irrelevant parameters)

(file k, file m, file n) Antchmbr (file l) {
        app { 
        antch "-i" @l ;
        }
}

If by chance k, m, and n happened to be mapped to submit-side paths that 
co-incide in the right way with the input path of l, then this happens to 
work. But in the general case, with k, m and n being mapped to arbitrary 
locations, this does not work, because at present swift assumes that the 
app is outputting to the (relative remote) paths k, m, n. This cannot be 
assumed, because that information about k,m,n is never passed to the 
application.


Here are a few potential solutions for 22. I'm interested in opinions on 
implementation ease, applicability to other peoples app problems that they 
have now, use to end users in general, and of course other solutions.

1. assert that remote end application-specific wrappers are the way to go, 
with those remote end wrappers remoulding the program interface to 
something more palateable to swift (in this case by moving input and 
output files around appropriately, on either side of an application 
invocation).

2. extend the SwiftScript app {} syntax to allow different expression of 
output files, by specifying that swift should expect to find them in more 
explicitly named files, rather than passing on the app commandline.

Thus we might change the above app code fragment to something like this:

 (file k, file m, file n) Antchmbr (file l) {
        app { 
            antch "-i" @l ;
            k <@strcat(@l,".crd");
            m <@strcat(@l,".prd");
            n <@strcat(@l,".rtf");
        }
 }

which says that when the application has run, the three output files will 
be located in/under the run directory in those locations. This would then 
give swift sufficient information to locate the three output files rather 
than assuming the output names.


My opinions:

sol1. I dislike the concept of remote wrappers from a users perspective - 
conceptually, app {} blocks are where we tell swift how to interface to 
programs. Using remote wrappers basically distributes that information 
between the app {} block and a remote wrapper; I find that unpleasant and 
would rather keep the information in one place - the app {} block.

sol2. more development work. from a user perspective I find it more 
pleasing because it keeps the interface description in one place, the app 
block. it introduces something that starts to look like mappers for the 
execution side of things - this might be an interesting longer term 
strategy for dealing with non-file-based data too; or might not - 
Yong/Mike probably have thought about the most so I'm interested in 
comment there.


Like I said at the top, I'm most interested in scoping 0.2 work, rather 
than long term stuff, but I want it to be coherent with the long term 
vision.

--