[Swift-user] XDTM
J A
jamalphd at gmail.com
Fri Aug 14 14:02:24 CDT 2009
Thanks for your help. I will analyze the data i have and let you know of
any updates.
On Fri, Aug 14, 2009 at 11:38 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:
> On 8/7/09 3:40 PM, J A wrote:
>
>> Hi Michael:
>>
>> 1. After running a .swift or .dtm code, two files gets created: .xml
>> and .klm. What do they represent?
>>
>
> .xml is an xml version of the parsed .swift file
> .kml (not klm) is the xml representation of the Karajan script that the
> Swift script is translated into for execution. Its actually the .kml file
> that is executed by Karajan which drives the execution of a Swift script.
>
> 2. Correct me if I am wrong:
>> * Datasets are mapped to physical presentation using mapping
>> algorithms. Some mapping algorithms already created part of
>> swift and the user can add/create others and use the
>> existing once as the base.
>>
>
> Yes, thats right.
>
> But, to clarify this part:
> "user can add/create others and use the existing ones as the base."
>
> The user can use existing mappers, and add new mappers, either in Java or
> as external executables or scripts. But each mapper is independent. When you
> say "can use existing ones as a base" I would say thats correct, in that a
> user could *copy* and modify the code of one mapper to create another
> mapper, or, in the case of an "ext" mapper, one ext mapper could conceivably
> execute another and modify/filter its output to create a new mapping.
>
> * Currently, the physical representation are files.
>>
>
> Yes, if you mean to say that mappers map files to Swift variables.
>
> 3. In the fMRI example, I see volume, Image, etc declared as
>> a type? who defines them as a type?
>>
>
>
> 4. In one of your emails, you stated that Swift functions can take
>> accept files, int, string, float and boolean values as arguments.
>> They return files, or scalar values inside files. My question is:
>> if the output is a string that is inside a file, how can I use
>> this output in another program that takes it as an input? doesn't
>> call the file name and should have a code to read from the file?
>>
>
> Yes, you can use readData() or readData2() to read the contents of a file
> back into Swift variables (including into arrays and structures, if the
> output has some structure).
>
> 5. I am still confused when talk about XML Data Type and Mapping.
>> Where is the XML representation? Is it the .xml that gets
>> generated when run the swift code?
>>
>
> No, the XML - if indeed it still exists - is only internal. I described it
> this way in an earlier post:
>
> --
>
> "As Swift evolved from its early prototypes to a more mature system, the
> notion of XDTM evolved to one of mapping between filesystem-based structures
> and Swift in-memory data structures (ie, scalars, arrays, and structures,
> which can be nested and typed).
>
> This is best seen by looking at the "external" mapper, ...
>
> In other words, it still has the flavor of XDTM, but without any XML being
> visible to the user. It meets the same need but is easier to use and
> explain."
>
> --
>
> When XDTM was first implemented, by Yong Zhao, he used XML within Swift to
> represent the mapping. I am not even sure if this XML representation is
> still used in the current implementation, or not. I suspect *not*.
>
> But the important concept here should really be called "DTM" - dataset
> typing and mapping - and its embodied in the type model and mapping model of
> the language.
>
> So you should stop thinking about data typing and mapping as being
> connected in any way to XML.
>
> What we described in earlier papers as XDTM is not something that you can
> experiment with in terms of XML: ie, you can not see the XML for a mapping
> because its either deep inside the Swift implementation, or it no longer
> exists in the current Swift code.
>
> 6. Let's look at this example:
>>
>> type messagefile {}
>> (messagefile t) greeting (string s[]) { app {
>> echo s[0] s[1] s[2] stdout=@filename(t
>> <mailto:stdout=@filename(t>);
>> }
>> }
>> messagefile outfile <"q5out.txt">;
>> string words[] = ["how","are","you"];
>> outfile = greeting(words);
>> ===
>> So we have messagefile as a data type. outfile and words are
>> datasets. what will be the physical representation for these 2
>> datasets?
>>
>
> An object of type messagefile will be represented as a single physical file
> externally, and internally as a scalar variable.
>
> Words is a an array of strings.
>
> Each atomic Swift variable (ie, scalars, array members, and structure
> members) can be thought of as a triple:
>
> (set-state, mapping, value)
>
> All variables have a set-state; initially unset, then set when the variable
> is assigned a value.
>
> File-valued variables have only a mapping, but no value.
> Scalar-values (ie, non-mapped variables like strings, as in your example)
> have a value (eg the string, interger, boolean or float value) but no
> mapping.
>
> We're still looking for better terminology to describe this; the current
> user guide uses both the terms "mapped type" and "marker type" to denote a
> file-valued variable. Both terms refer to the same concept; Im leaning to
> the term "mapped type".
>
> is thee system parsing the swift code, identifying the
>
>> data types and datasets and based on that choosea the proper mapping
>> algorithm needed?
>>
>
> After the Swift command parses the Swift code, execution begins - i.e. the
> .kml file is executed by Karajan. Mappers are called as can be seen the kml.
> (And you can see their actions in the swift .log file).
>
> The mapping for all mapped variables is either specified by the user (the
> most common case) or defaults to concurrent_mapper.
>
> The users guide describes this in pretty good detail.
>
> I hope that gets you a bit further. I hope that looking at XML mappings is
> not critical to your research, as I don't think you'll be able to readily
> get an XML intermediate form out of Swift.
>
> An interesting topic would be to implement mechanisms to handle data in XML
> representations, in particular to enable Swift to invoke SOAP services as
> well as file-based applications and to compose scripts that call both forms
> of application.
>
> - Mike
>
>
>> Thanks,
>> Jamal
>>
>>
>>
>>
>>
>> On Sun, Jul 26, 2009 at 9:53 PM, Michael Wilde <wilde at mcs.anl.gov<mailto:
>> wilde at mcs.anl.gov>> wrote:
>>
>> Hi Jamal,
>>
>> A lot of this is covered in the Swift user guide and tutorial. Have
>> you read through those yet?
>>
>> All the docs are at: http://www.ci.uchicago.edu/swift/docs/index.php
>>
>> If so, and the clarifications below don't help, please ask again on
>> the list, OK?
>>
>> - Mike
>>
>>
>>
>> On 7/26/09 7:27 PM, J A wrote:
>>
>> Hi Michael:
>> First, thank you for your reply and information provided.
>> I am trying to understand more how it handles the input/output
>> parameters and make them available for other functions.
>>
>>
>> All functions in Swift are either atomic interfaces to application
>> programs (ie, how o exec the program) or composite higher level
>> functions.
>>
>> To illustrate, I will give this example for the sake of
>> discussion:
>> I have a C program called test.c that contains 4 functions (
>> main(), F1, F2, and F3). each function takes some parameters
>> such as int, string, name of a file that is in the same
>> directory, and each one produced some output (string, int, and a
>> file). Of course i am using global variables. Now, main calls
>> F1, F1 passes its output to F2, and F2 passes its output to F3.
>>
>>
>> Swift doesnt look at the functions inside an application. It invokes
>> the application as a program (think fork/exec) just like a shell
>> would, but distributed and in parallel if so specified.
>>
>> Overall, the test.c takes an int, string, and file, and output
>> several files. the output files contains output produced by the
>> internal functions (tasks).
>>
>>
>> Swift functions can take accept files, int, string, float and
>> boolean values as arguments. They return files, or scalar values
>> inside files. (Again, think shell scripts). Composite structures -
>> structs and arrays - of the above can be passed.
>>
>> I would like to understand more when i transfer my code to
>> Swift how it handles the input/output data, where it stores
>> them, etc. I read couple of papers about XDTM and still have
>> some confusion about the terms: dataset, typed, how/where its
>> physical representation is located at, and how the input/output
>> is used within the internal functions.
>>
>>
>> Files are by default named ("mapped") relative to the directory in
>> which you run the Swift command. Many flexible extensions to that
>> model are provided for (eg, URIs). Swift sends the data to the site
>> chosen for execution (thats yet another topic) and returns results
>> back to the same submission host.
>>
>> Mapping declarations in the Swift script specify how files and
>> directory structures are mapped to Swift variables (scalars, arrays,
>> structures). These are used in the specification of the Swift code.
>> When Swift runs programs, it takes files that were mapped and knows
>> how to send them to grid sites or clusters and get data back.
>>
>> I am new to this area and trying to understand how the DTM works.
>> Any help from your side on this area is really appreciated.
>> Thanks,
>> Jamal
>> On Sun, Jul 26, 2009 at 7:09 PM, Michael Wilde
>> <wilde at mcs.anl.gov <mailto:wilde at mcs.anl.gov>
>> <mailto:wilde at mcs.anl.gov <mailto:wilde at mcs.anl.gov>>> wrote:
>>
>> Jamal,
>>
>> As Swift evolved from its early prototypes to a more mature
>> system,
>> the notion of XDTM evolved to one of mapping between
>> filesystem-based structures and Swift in-memory data
>> structures (ie,
>> scalars, arrays, and structures, which can be nested and typed).
>>
>> This is best seen by looking at the "external" mapper, which
>> allows
>> a user to map a dataset using any external program (typically a
>> script) that returns the members of the dataset as a two-column
>> list: the Swift variable reference, and the external file or
>> URI.
>>
>> See the user guide section on the external mapper:
>>
>>
>> http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.ext_mapper
>> (but the example in the user guide doesn't show the power of
>> mapping
>> to nested structures).
>>
>> In other words, it still has the flavor of XDTM, but without
>> any XML
>> being visible to the user. It meets the same need but is
>> easier to
>> use and explain.
>>
>> - Mike
>>
>>
>> On 7/26/09 2:50 PM, J A wrote:
>>
>> Hi All:
>> Can any one direct me to a source with more
>> examples/explanation on how XDTM is working/implemented?
>> Thanks,
>> Jamal
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> <mailto:Swift-user at ci.uchicago.edu>
>> <mailto:Swift-user at ci.uchicago.edu
>> <mailto:Swift-user at ci.uchicago.edu>>
>>
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20090814/e12dd18c/attachment.html>
More information about the Swift-user
mailing list