[Swift-devel] Re: passing types and variables in swift

[Michael Wilde]

I'm certainly not an expert swift coder, but here's what I came up with 
for what I think is your use case, Andrew.

The need as I understand is to associate a unique set of scalar 
parameters with each file (or set of files) that will be passed to an 
application.

This seems to call for an array of structs, where the struct fields have 
a mixture of file-values and scalar-values.  Then you do a foreach over 
the array of structs, and pass one element of the array (a struct) to 
each invocation of the app.

A csv mapper that allows scalar fields like Yong suggests would do it 
nicely, but I dont yet fully grok what that does to the data model.

While people debate this, you can with a tiny script (sh, awk, perl) 
turn a CSV file into the set of swift statements between the AUTOGEN 
comments below, and then include them into your swift script before you 
pass it to the swift command.

- Mike

type messagefile;

type file;

type params {
   string x;
   string y;
   file f;
};

(messagefile t) pecho (params p ) {
     app {
         echo p.x p.y @p.f stdout=@filename(t);
     }
}

/* START AUTOGEN */

/* map the file params this way */

string fn[] = ["f001","f002"];
file   fv[] <array_mapper;files=fn>;

/* then stuff file vals into a struct to associate with scalar vals */

params plist[];

plist[0].x = "x1111";
plist[0].y = "y2222";
plist[0].f = fv[0];

plist[1].x = "x1111";
plist[1].y = "y2222";
plist[1].f = fv[1];

/* etc */
/* END AUTOGEN */

messagefile outfile <"hello2.txt">;

/* then you can iterate over the param sets and pass each set as a
struct */

foreach pval in plist {
   outfile = pecho(pval);
}


On 10/4/07 9:48 AM, andrewj at uchicago.edu wrote:
> Mike,
> 
>   Your question does make sense.  I am looking for the most
> general model and configurable model which would allow us to
> not have to mess with the swift script as much as possible,
> but rather simply be able to dump a set of parameter setting
> files som,ewhere and swift pick them up and arrange things
> based on this.
> 
> And yes, this is due in part to the concern of very long
> lists. But maybe I am over thinking things for right now. 
> Maybe it will work out just as efficiently to have mulitple
> swift WF codes with the different param settings.
> 
> I think however for the time being I will create the work flow
> to just use the parameter settings in the Swift code itself.
> 
> Thanks,
> Andrew
> 
> ---- Original message ----
>> Date: Thu, 04 Oct 2007 09:24:51 -0500
>> From: Michael Wilde <wilde at mcs.anl.gov>  
>> Subject: Re: passing types and variables in swift  
>> To: andrewj at uchicago.edu
>> Cc: Ben Clifford <benc at hawaga.org.uk>, Mihael Hategan
> <hategan at mcs.anl.gov>, swift-devel at ci.uchicago.edu
>> I'll try to push forward with this as Ben is focusing on SC
> tutorials today.
>> Andrew, lets try to make a simple model of your workflow that
> we can 
>> actually run in local mode, using mock apps.
>>
>> I need to go backwards, but basically this is a parameter
> sweep, where 
>> the parameter sets have a nested loop structure. Im noty sure
> f Im 
>> asking this right (as Im in a meeting and cant read right
> now) but do 
>> you want to express the parameters in files (because the
> lists are long) 
>> or right in the source code of the workflow?  Does that
> question make sense?
>> - Mike
>>
>>
>> On 10/4/07 9:16 AM, andrewj at uchicago.edu wrote:
>>> All right, thanks.
>>>
>>> I am just trying to figure out the best way to setup parameter
>>> sweeps.  Ideally, I would like to have some file of directory
>>> of files containing experiments to be run on the Grid
>>> environment and Swift picks them up and automatically sets up
>>> the appropriate WF based on the parameter information in the
>>> files.
>>>
>>>
>>>> On Wed, 3 Oct 2007, andrewj at uchicago.edu wrote:
>>>>
>>>>> could I do something like this:
>>>>>
>>>>> RGIparams RGIinput[] <csv_mapper;file="rgi_runs.txt">;
>>>>>
>>>>> where rgi_runs would contain say
>>>>>
>>>>> 1,2,true
>>>>> 3,4,false
>>>>> 18,20,true
>>>>>
>>>>> and so on..so that the parameters are passed into the
> workflow?
>>>> no.
>>>>
>>>> mappers only map filenames (or URIs now); they don't map
>>> variable values.
>>>> -- 
>>>
> 
>