[Swift-devel] Re: passing types and variables in swift

[Michael Wilde]

Thanks for all the details - its helping to clarify the mystery of 
mappers. But I dont get all this on a quick read so I need to study it. 
Seems like we need to fill in a few more details of the end-user-visible 
parts of this model beyond what is in the current tutorial and UG, but 
less than what you explain here.

Dont know how DMA got into the picture, but I need to stew on this for a 
while.

- Mike


On 10/4/07 11:27 PM, Mihael Hategan wrote:
> On Thu, 2007-10-04 at 23:06 -0500, Mihael Hategan wrote:
>> On Thu, 2007-10-04 at 22:53 -0500, Yong Zhao wrote:
>>> Mike
>>>
>>>> while we're discussing mappers, could you or Ben clarify:
>>>>
>>>> - can a mapper be only used on a single variable declaration? ie you can
>>>> not map a dataset to a variable within a struct?  this would be useful
>>>> eg to create a struct that has both file and scalar valued members, to
>>>> set up a parameter sweep where each file needs specific parameters.
>>> This would be the nested mapper scenario, in order to make the scripts
>>> cleaner, I was think about a mapping descriptor (which could be an XML
>>> file to describe each layer of the mapping), but this is kind of far
>>> reaching.
>> I think C pointers provide a very good model for this. We should perhaps
>> use that as a reference.
> 
> So let me elaborate on that.
> 
> All non-primitive types are pointers. For simplicity, we should have all
> types as pointers, but currently we distinguish a bit between primitive
> and non-primitive.
> 
> A complex type variable in Swift is like a pointer to a struct of
> pointers.
> 
> In the beginning, Swift needs to figure out the exact addresses of those
> pointers. That's what the existing() mapper method does. It goes through
> the struct and recursively initializes the pointer addresses based on
> some scheme which depends on the mapper implementation. This is like
> doing a recursive field = malloc(sizeof(field)).
> 
> The map method of a mapper takes a struct path (a.b.c) and returns the
> address.
> 
> Currently, nothing ever uses a value of a pointer. Grid applications are
> passed the address (file name).
> 
> Getting/setting values amounts to pointer dereferencing. Both read and
> write.
> 
> The addresses of these pointers are abstract. They don't represent
> anything in particular. Currently we assume that they are specific
> addresses (files), but desire is to extends this to databases and
> things. Let's say that we have a segmented address space and our
> pointers can be in different segments.
> 
> The segment issue only applies to how we read/write values.
> 
> There is one plain memory segment. Currently only primitive types can be
> here.
> 
> There may be multiple memory-mapped I/O segments. Say one for files, one
> for databases, etc. We need to implement what actually happens when you
> read/write from/to a I/O segment. Basically mappers supporting
> read/write of values are the I/O hardware drivers.
> 
> Whenever moving data between segments, we are either efficient and use
> DMA transfers, or we read to the plain memory segment and then write
> back to the other I/O segment. This is the analogy of, for example,
> getting some value from a database to a file so it can be passed to an
> application. The DMA case is when there is some entity that knows how to
> convert directly between the two. The non-DMA case is when we convert
> first to a common format and then back to another format.
> 
>>>> - are mappers processed before the program (or procedure) starts
>>>> executing? so you cant use a mapper like an assignment statement to set
>>>> or reset a variable?
>>>>
>>> Mappers are evaluated before procedure execution, the only exception is
>>> when a mapper depends on some intermediate data, and it will wait for that
>>> data to be available (as in the montage workflow). However, mapping is
>>> different from assignment (in assignment, it is more like calling a
>>> mapping function such as @extractint). A variable currently can not be
>>> reset or reassigned, unless it is an interation variable (it gets
>>> re-assigned implicitly).
>>>
>>> Yong.
>>>
>>>> - mike
>>>>
>>>> On 10/4/07 10:21 PM, Yong Zhao wrote:
>>>>> Mappers should ideally be able to map primitive types such as string and
>>>>> int, as well as file names. The CSVMapper can read all the values in the
>>>>> file, it is just that it needs to interprete the values according to their
>>>>> types. So If we convey the type info 'RGIparams' to the csv mapper, it
>>>>> should have no problem reading the values.
>>>>>
>>>>> I actually added getType and setType to the mapper interface, that did
>>>>> not get into the release, but I think it is what it should have.
>>>>>
>>>>> Yong.
>>>>>
>>>>> On Thu, 4 Oct 2007 andrewj at uchicago.edu wrote:
>>>>>
>>>>>> All right, thanks.
>>>>>>
>>>>>> I am just trying to figure out the best way to setup parameter
>>>>>> sweeps.  Ideally, I would like to have some file of directory
>>>>>> of files containing experiments to be run on the Grid
>>>>>> environment and Swift picks them up and automatically sets up
>>>>>> the appropriate WF based on the parameter information in the
>>>>>> files.
>>>>>>
>>>>>>
>>>>>>> On Wed, 3 Oct 2007, andrewj at uchicago.edu wrote:
>>>>>>>
>>>>>>>> could I do something like this:
>>>>>>>>
>>>>>>>> RGIparams RGIinput[] <csv_mapper;file="rgi_runs.txt">;
>>>>>>>>
>>>>>>>> where rgi_runs would contain say
>>>>>>>>
>>>>>>>> 1,2,true
>>>>>>>> 3,4,false
>>>>>>>> 18,20,true
>>>>>>>>
>>>>>>>> and so on..so that the parameters are passed into the workflow?
>>>>>>> no.
>>>>>>>
>>>>>>> mappers only map filenames (or URIs now); they don't map
>>>>>> variable values.
>>>>>>> --
>>>>>> _______________________________________________
>>>>>> Swift-devel mailing list
>>>>>> Swift-devel at ci.uchicago.edu
>>>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>>>>>
>>>>> _______________________________________________
>>>>> Swift-devel mailing list
>>>>> Swift-devel at ci.uchicago.edu
>>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>>>>
>>>>>
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
>