[Swift-devel] remote file/directory stuff (bug 22) (fwd)

[Ben Clifford]

there was a thread a couple of months (almost to the day) about this 0.2 
feature.

I've done some work towards implementing this already, and I'm going to do 
some more now.

I'm planning on implementing the syntax Yong discusses below. I hit the 
problem that motivated this again yesterday, working with Nika.

I'm not sure what remote mapping should really look like - pretty much 
everyone seems in agreement about the general concept, with various 
differences, so I think it will be useful to actually have an 
implementation of *something* to get a practical feel, rather than our 
previous interminable discussion threads.

---------- Forwarded message ----------
Date: Sat, 3 Mar 2007 01:37:42 +0000 (GMT)
From: Ben Clifford <benc at hawaga.org.uk>
To: swft at ci.uchicago.edu
Subject: [Swft] Re: [Swift-devel] remote file/directory stuff (bug 22)


So the message below can be the beginnings of campaign definition for bug 
22 - 'execute-side protomappers'.

I think in terms of work, me (or Yong if he wants to) needs to implement 
the swiftscript->kml bit to generate the kml syntax that Mihael suggested 
in an earlier message; and then Mihael should go from there to make it 
happen inside the execution engine.

On Fri, 2 Mar 2007, Ben Clifford wrote:

> 
> 
> On Fri, 2 Mar 2007, Yong Zhao wrote:
> 
> > Can you elaborate on this issue a little bit so that we can make a
> > unanimous decision:
> > 
> > 1. what was the problem exactly
> 
> Some programs that we run in swift do not use the traditional VDS-like API 
> of being told on the commandline the names of the files that they must 
> input and output to. Instead, they make up some of the names themselves.
> 
> For example, one of Nika's programs has the syntax:
> 
>  ./program inputfilename
> 
> and places its outputs in inputfile.stuff, inputfile.abc, inputfile.foo
> 
> > 2. what are you proposing
> 
> To extend the syntax of the app {} block to permit specification of the 
> above interface, with a syntax something like:
> 
> (stuffoutfile s, abcoutfile a, foooutfile f) myproc(inputfile i)
>   app { 
>     program @i;
>     s < @strcat(@inputfile,".stuff")
>     a < @strcat(@inputfile,".abc")
>     f < @strcat(@inputfile,".foo")
>   }
>  }
> 
> Meaning that rather than Swift specifying the remote name for s, a and f, 
> instead the app block specifies where those three files are.
> 
> These will be staged back into the submit-side location defined in the 
> existing mappers.
> 
> > 3. to what extent does the proposal solve the problem
> 
> It should solve Nika's immediate problem, I think.
> 
> > 4. what is the implication to the mapping interface
> 
> A longer term perspective is that this is the beginning of longer work to 
> implement fuller execute-side mappers (which have also been called 
> application mappers in some threads).
> 
> So it is mapping, but on the execute side. It fits in in a fairly 
> straightforward way with mapping on the submit side, which is what we have 
> now.
> 
> Submit side mapping maps between submit-side data and SwiftScript 
> variables/structures, so that the user can arrange his submit-side data in 
> a way that he wants (rather than swift compelling it to be in a particular 
> format)
> 
> Execute side mapping maps between SwiftScript variables and execute side 
> data, so that data can be laid out on the execute side in the way that the 
> program wants it (rather than swift compelling it to be in a particular 
> format)
> 
> With the present implementation, this amounts to being able to specify 
> different paths and filenames on the submit and execute side for each data 
> file.
> 
> In the longer term, it might also be useful in defining things like how to 
> map data on a submit-side database to some format on the execute side for 
> processing. If we have only submit side mappers, then we can map data 
> between a submit side database and SwiftScript structures, but not map 
> between those structures and the execute side...
> 
> --
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
>