[Swift-devel] Re: Phone meeting?

[Mihael Hategan]

On Tue, 2008-01-15 at 11:16 -0600, Jim Kowalkowski wrote:
> Hi Mihael,
> 
> See below for comments (from all of us).
> You can forward this to your development list, since I don't think I can 
> post directly to it.

That's ok. Somebody will get a bounce and they will approve the post.

> 
> Jim (for the LQCD group)

> >
> > app (quark output) cvt12x12 (file input) {
> >         CVT12x12 "-v" "-i" @input "-o" @output;
> > }
> >   
> Cosmetic changes like these make the script easier to understand and we 
> will be able to convince the
> domain scientists that its lot easier for them to adopt it.

I thought it would make more sense for reasons beyond cosmetics. The
app{} block cannot, for example, be used in a foreach loop (or at least
it isn't well defined there). So it is bound to a "procedure"
definition.

> 
> One minor issue - would it be better to place this "function attribute" 
> closer to the function name,
> such as "(quark output) app cvt12x12 (file input)"? Or perhaps after the 
> function name?

I'd personally like to follow known styles (e.g. Java), but I guess the
parser can be made to understand "app" flexibly.

> > 2. Improved iteration. I already sent an email to swift-devel, but I'll
> > re-iterate here:
> > (a) for (x, y) in (X, Y) {} will be roughly equivalent to
> > and
> > (b) for (x, y) in (X * Y) {} will be equivalent to:
>
> Again, cosmetic changes like this are useful.  And we agree that (a) is 
> more useful than (b).
> 
> > 3. I'll need a list of data objects for which you care about the file
> > names, and the corresponding file names.
> >   
> 
> Coming up with an answer for questions (3) and (4) was somewhat 
> difficult. We thought that it would be
> best  for us to show you an example that demonstrates the Swift language 
> features that we are thinking about.
> This also addresses some of the issues involved in (4), essentially 
> because it does away with the need to
> reference descriptive information from more than one place (i.e. 
> multiple arrays and file objects).

... in Swift.
However, lots of things are now pushed into the mappers, so the
referencing of descriptive information from multiple places may still
happen. Or not.

> Hopefully this will be more clear in the example below than in the short 
> description above.
> 

> /**
> * This workflow was created based on a portion of the 2ptHL SwiftScript.
> * Many steps are not implemented here, but it does illustrates the
> * desired functionality of assigning mappers to types. This eliminates
> * the nested loops and complex array indices.
> */
> 
> /**
> * Types can be bound to mappers at declaration.
> */
> type ParameterSet<ParameterSetMapper>; type HeavyQuark<HeavyQuarkMapper>;
> type HeavyQuarkConv<HeavyQuarkConvMapper>;
> type TwoPointHLFile<TwoPointHLMapper>;

Interesting. So declarations will then parametrize those mappers.
I think this needs more thinking. As you point out below, there are some
issues.

> 
> /**
> * The set of parameters is retrieved from a database (or file system) using
> * the specified key. The parameter come in as name-value pairs.  The values
> * are attached to the object as attributes available using the name.
> */
> 
> /* in our Dec discussion, this was referred to as "parameter 57" */
> ParameterSet pset<id="MyParameterSetKey">;
> 
> /**
> * The parameter set values are used to create a set of file names - heavy
> * quarks in this example. In this case, the mapper bound to HeavyQuark 
> accesses
> * the wsrc attribute to make the files.  The [] mean that the variable is a
> * container (which might be multidimensional internally), it is not an 
> array,
> * but a object with some array-like semantics.
> *
> * We are leaving out some important details here - one is that the 
> mapper is bound
> * to a HeavyQuark, not a container of HeavyQuarks.
>   This could mean that 
> mapper must
> * know how to work with containers of the type they are bound to i.e. 
> have some sort
> * of container operator defined.

Also, should one construct a struct with fields of HeavyQuark type,
there is some blurriness in how things would be done.

> *
> * We are also assuming a very simple structure to the parameter set.  A 
> more
> * interesting scenario would be some hierarchy within the parameter set, 
> in other
> * words:  pset at 2pt would give the child parameter set object
> * specific to TwoPointHLFiles.
> */
> HeavyQuark heavyQuarkFiles[]<params=pset>;
> HeavyQuarkConv heavyQuarkConvertedFiles[]<params=pset>;
> TwoPointHLFile twoPointHLFile[]<params=pset>;
> 
> /**
> * Create the heavy quark file from the parameter set. We specify the 
> parameter
> * set as input, but the mapper attaches the attributes (e.g. kappa) to the
> * object. So 'out' is really an object here, where all the attributes that
> * went into making the filename are also individually available.  This 
> eliminates
> * the need for passing in data that was already used in the creation of
> * the filename i.e. removes redundancy and need of extra looping at the 
> higher
> * level.

So in principle file names as well as data depend on both parameters
specified in the script, as well as input data.

However, @ is weird here. I'm thinking that this can be done fairly well
with structs. For example:

type HeavyQuark {
  ParameterSet pset;
  HQData data;
}

and then a constructor:
(HeavyQuark quark) HeavyQuark(ParameterSet pset) {
  quark.pset = pset;
  quark.data = CreateHQ(pset);
}

app (HQData data) CreateHQ(ParameterSet pset) {
  wrapper "-k" pset.kappa "-w" pset.csw "-o" filename(data);
}

app (HeavyQuarkConv out) HQConv(HeavyQuark q) {
  wrapper "-k" q.pset.kappa "-w" q.pset.csw "-o" filename(out);
}

This would avoid a lot of complexity in how the parameter set would be
magically and not quite clearly moved around.

Ok, I'm a bit fuzzy on kappa. It's attached to the quark, but it's
supposed to be there before the quark is there.

> */
> app (HeavyQuark out) CreateHQ (ParameterSet pset) {
>  wrapper -k out at kappa -w pset at csw -o @filename(out);
> }
> 
> app (HeavyQuarkConv out) HQConv (HeavyQuark q) {
>  wrapper -i q at kappa -o out;
> }
> 
> /**
> * The parameters lq and wf are not define explicitly in this example,
> * but are similar to hq.
> * We need extra functionality to extract a file name from a set based on 
> a list
> * of attributes, that is what hq.query () and lq.find () are trying to do.
> * This adds random access to the container, linking it to the mapper, 
> instead
> * of just assuming index-only access.
> * Not all attributes used for finding lq and hq files are used in this 
> example.
> */
> app (TwoPointHLFile hlFile) TwoPointHL (ParameterSet pset, 
> HeavyQuarkConv[] hq,
>                                        LightQuark[] lq, WaveFile[] wf) {
> 
>  wrapper -o @filename (hlFile) -h hq.query (hlFile at kappa) \
>          -l lq.find (hlFile at kappa, hlFile at mass, hlFile at d1)
> }

So there are two choices here:
1. These query functions are written in Swift (perhaps unwise at this
time)
2. The query functions are written in something else. In fact, they
could be mappers:

HeavyQuarkConv[] hqSubset1 <queryMapper;source=hq, kappa=pset.kappa>;
...

Mihael

> 
> /* creation only depends on the parameters in the parameter set */
> foreach qf in heavyQuarkFiles {
>  qf = CreateHQ (pset);
> }
> 
> /* conversion requires us to move two iterators at the same time */
> foreach qf,qfout in heavyQuarkFiles, heavyQuarkConvertedFiles {
>  qfout = HQConv (qf);
> }
> 
> /**
> * This replaces the last set of nested loops of the 2ptHL workflow.
> * The file object contain all the information necessary to do the
> * processing.
> */
> foreach hl in twoPointHLFile {
>  hl = TwoPointHL (heavyQuarkConvertedFiles, lightQuarkFiles, waveFiles);
> }
> 
>