[Swift-devel] [Bug 76] disable intermediate stageout of data

[Mihael Hategan]

On Thu, 2007-07-05 at 12:14 -0500, Yong Zhao wrote:
> My original thinking about value/filename was that we don't distinguish
> those at the logical level, essentially they could all just be values.
> Then when we need to call mapper functions (getFilename, for instance), we
> interprete the values differently. So in the case of getFilename, we can
> interprete the value either as
> 1) the filename itself, returning the value directly
> 2) writing the value into a file, and returning an automatically generated
> filename.
> 3) some other possibilities, e.g. a directory of files.

This clearly conflicts with the ability to apply swift functions to data
in files or databases, as one would need, in the case of files, both a
file pointer and actual data.

I would rather follow a known model for this: pointers. There are
addresses (files, uris, db/table/column/row) and values, which are
stored at those addresses. What's missing from the scheme right now is
the ability of a mapper to fetch actual data from such locations when
needed.

> 
> The current DSHandle interface does allow nested trees, so a mapper
> could return a dshandle tree as the implementation currently stands.
> 
> Yong.
> 
> 
> On Thu, 5 Jul 2007, Ben Clifford wrote:
> 
> >
> > so I was thinking the other day while poking through code.
> >
> > 'data' in SwiftScript terms is mostly represented by DSHandle objects.
> >
> > such objects (which can have one of several implementing classes, and
> > potentially more in future) have a number of properties, such as:
> >
> >  .  value - what the 'value' is, for adding to other values, using @strcat
> > on, performing array/member access using [] and .
> >
> >  . submit-side location - what is extracted with @filename and used when
> > that 'data' is passed to an application rather than being operated on by
> > submit-side functions.
> >
> > Neither of these are compulsory (and I think in practice at the moment it
> > works out that you either have a filename or a value and never
> > meaningfully both).
> >
> > So a different model of mapping (which might work better when we want data
> > that doesn't necessarily exist as discrete files or as in-memory values -
> > the two examples that I've seen talked about are 'data from an sql
> > database' and 'constants in a csv file') might be that mappers generate
> > DSHandle trees (specifically a mapper generates a DSHandle, which might
> > have descendants). Those DSHandles might have values, might have
> > filenames, might have other attributes, might have ongoing annotation
> > (which could include keeping track of where within-this-run copies have
> > been made).
> >
> > --
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >
>