[Swift-devel] remote file/directory stuff (bug 22)

[Ben Clifford]

On Fri, 2 Mar 2007, Yong Zhao wrote:

> Can you elaborate on this issue a little bit so that we can make a
> unanimous decision:
> 
> 1. what was the problem exactly

Some programs that we run in swift do not use the traditional VDS-like API 
of being told on the commandline the names of the files that they must 
input and output to. Instead, they make up some of the names themselves.

For example, one of Nika's programs has the syntax:

 ./program inputfilename

and places its outputs in inputfile.stuff, inputfile.abc, inputfile.foo

> 2. what are you proposing

To extend the syntax of the app {} block to permit specification of the 
above interface, with a syntax something like:

(stuffoutfile s, abcoutfile a, foooutfile f) myproc(inputfile i)
  app { 
    program @i;
    s < @strcat(@inputfile,".stuff")
    a < @strcat(@inputfile,".abc")
    f < @strcat(@inputfile,".foo")
  }
 }

Meaning that rather than Swift specifying the remote name for s, a and f, 
instead the app block specifies where those three files are.

These will be staged back into the submit-side location defined in the 
existing mappers.

> 3. to what extent does the proposal solve the problem

It should solve Nika's immediate problem, I think.

> 4. what is the implication to the mapping interface

A longer term perspective is that this is the beginning of longer work to 
implement fuller execute-side mappers (which have also been called 
application mappers in some threads).

So it is mapping, but on the execute side. It fits in in a fairly 
straightforward way with mapping on the submit side, which is what we have 
now.

Submit side mapping maps between submit-side data and SwiftScript 
variables/structures, so that the user can arrange his submit-side data in 
a way that he wants (rather than swift compelling it to be in a particular 
format)

Execute side mapping maps between SwiftScript variables and execute side 
data, so that data can be laid out on the execute side in the way that the 
program wants it (rather than swift compelling it to be in a particular 
format)

With the present implementation, this amounts to being able to specify 
different paths and filenames on the submit and execute side for each data 
file.

In the longer term, it might also be useful in defining things like how to 
map data on a submit-side database to some format on the execute side for 
processing. If we have only submit side mappers, then we can map data 
between a submit side database and SwiftScript structures, but not map 
between those structures and the execute side...

--