[Swift-devel] Re: passing types and variables in swift

Mihael Hategan hategan at mcs.anl.gov
Thu Oct 4 23:45:07 CDT 2007 

On Thu, 2007-10-04 at 23:37 -0500, Michael Wilde wrote:
> Thanks for all the details - its helping to clarify the mystery of 
> mappers. But I dont get all this on a quick read so I need to study it. 
> Seems like we need to fill in a few more details of the end-user-visible 
> parts of this model beyond what is in the current tutorial and UG, but 
> less than what you explain here.
> 
> Dont know how DMA got into the picture,

It's an analogy. You can do funny things with it. Like transfer from HDD
to graphics card directly and such. It's efficient. The same applies in
the swift case.

>  but I need to stew on this for a 
> while.

It won't get much more familiar than this. We must start with some
common assumptions.

> 
> - Mike
> 
> 
> On 10/4/07 11:27 PM, Mihael Hategan wrote:
> > On Thu, 2007-10-04 at 23:06 -0500, Mihael Hategan wrote:
> >> On Thu, 2007-10-04 at 22:53 -0500, Yong Zhao wrote:
> >>> Mike
> >>>
> >>>> while we're discussing mappers, could you or Ben clarify:
> >>>>
> >>>> - can a mapper be only used on a single variable declaration? ie you can
> >>>> not map a dataset to a variable within a struct?  this would be useful
> >>>> eg to create a struct that has both file and scalar valued members, to
> >>>> set up a parameter sweep where each file needs specific parameters.
> >>> This would be the nested mapper scenario, in order to make the scripts
> >>> cleaner, I was think about a mapping descriptor (which could be an XML
> >>> file to describe each layer of the mapping), but this is kind of far
> >>> reaching.
> >> I think C pointers provide a very good model for this. We should perhaps
> >> use that as a reference.
> > 
> > So let me elaborate on that.
> > 
> > All non-primitive types are pointers. For simplicity, we should have all
> > types as pointers, but currently we distinguish a bit between primitive
> > and non-primitive.
> > 
> > A complex type variable in Swift is like a pointer to a struct of
> > pointers.
> > 
> > In the beginning, Swift needs to figure out the exact addresses of those
> > pointers. That's what the existing() mapper method does. It goes through
> > the struct and recursively initializes the pointer addresses based on
> > some scheme which depends on the mapper implementation. This is like
> > doing a recursive field = malloc(sizeof(field)).
> > 
> > The map method of a mapper takes a struct path (a.b.c) and returns the
> > address.
> > 
> > Currently, nothing ever uses a value of a pointer. Grid applications are
> > passed the address (file name).
> > 
> > Getting/setting values amounts to pointer dereferencing. Both read and
> > write.
> > 
> > The addresses of these pointers are abstract. They don't represent
> > anything in particular. Currently we assume that they are specific
> > addresses (files), but desire is to extends this to databases and
> > things. Let's say that we have a segmented address space and our
> > pointers can be in different segments.
> > 
> > The segment issue only applies to how we read/write values.
> > 
> > There is one plain memory segment. Currently only primitive types can be
> > here.
> > 
> > There may be multiple memory-mapped I/O segments. Say one for files, one
> > for databases, etc. We need to implement what actually happens when you
> > read/write from/to a I/O segment. Basically mappers supporting
> > read/write of values are the I/O hardware drivers.
> > 
> > Whenever moving data between segments, we are either efficient and use
> > DMA transfers, or we read to the plain memory segment and then write
> > back to the other I/O segment. This is the analogy of, for example,
> > getting some value from a database to a file so it can be passed to an
> > application. The DMA case is when there is some entity that knows how to
> > convert directly between the two. The non-DMA case is when we convert
> > first to a common format and then back to another format.
> > 
> >>>> - are mappers processed before the program (or procedure) starts
> >>>> executing? so you cant use a mapper like an assignment statement to set
> >>>> or reset a variable?
> >>>>
> >>> Mappers are evaluated before procedure execution, the only exception is
> >>> when a mapper depends on some intermediate data, and it will wait for that
> >>> data to be available (as in the montage workflow). However, mapping is
> >>> different from assignment (in assignment, it is more like calling a
> >>> mapping function such as @extractint). A variable currently can not be
> >>> reset or reassigned, unless it is an interation variable (it gets
> >>> re-assigned implicitly).
> >>>
> >>> Yong.
> >>>
> >>>> - mike
> >>>>
> >>>> On 10/4/07 10:21 PM, Yong Zhao wrote:
> >>>>> Mappers should ideally be able to map primitive types such as string and
> >>>>> int, as well as file names. The CSVMapper can read all the values in the
> >>>>> file, it is just that it needs to interprete the values according to their
> >>>>> types. So If we convey the type info 'RGIparams' to the csv mapper, it
> >>>>> should have no problem reading the values.
> >>>>>
> >>>>> I actually added getType and setType to the mapper interface, that did
> >>>>> not get into the release, but I think it is what it should have.
> >>>>>
> >>>>> Yong.
> >>>>>
> >>>>> On Thu, 4 Oct 2007 andrewj at uchicago.edu wrote:
> >>>>>
> >>>>>> All right, thanks.
> >>>>>>
> >>>>>> I am just trying to figure out the best way to setup parameter
> >>>>>> sweeps.  Ideally, I would like to have some file of directory
> >>>>>> of files containing experiments to be run on the Grid
> >>>>>> environment and Swift picks them up and automatically sets up
> >>>>>> the appropriate WF based on the parameter information in the
> >>>>>> files.
> >>>>>>
> >>>>>>
> >>>>>>> On Wed, 3 Oct 2007, andrewj at uchicago.edu wrote:
> >>>>>>>
> >>>>>>>> could I do something like this:
> >>>>>>>>
> >>>>>>>> RGIparams RGIinput[] <csv_mapper;file="rgi_runs.txt">;
> >>>>>>>>
> >>>>>>>> where rgi_runs would contain say
> >>>>>>>>
> >>>>>>>> 1,2,true
> >>>>>>>> 3,4,false
> >>>>>>>> 18,20,true
> >>>>>>>>
> >>>>>>>> and so on..so that the parameters are passed into the workflow?
> >>>>>>> no.
> >>>>>>>
> >>>>>>> mappers only map filenames (or URIs now); they don't map
> >>>>>> variable values.
> >>>>>>> --
> >>>>>> _______________________________________________
> >>>>>> Swift-devel mailing list
> >>>>>> Swift-devel at ci.uchicago.edu
> >>>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >>>>>>
> >>>>> _______________________________________________
> >>>>> Swift-devel mailing list
> >>>>> Swift-devel at ci.uchicago.edu
> >>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >>>>>
> >>>>>
> >>> _______________________________________________
> >>> Swift-devel mailing list
> >>> Swift-devel at ci.uchicago.edu
> >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >>>
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >>
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> > 
> > 
>

More information about the Swift-devel mailing list