<div><br>Thanks for your help. I will analyze the data i have and let you know of any updates.</div>
<div> </div>
<div> </div>
<div><br> </div>
<div class="gmail_quote">On Fri, Aug 14, 2009 at 11:38 AM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div class="im">On 8/7/09 3:40 PM, J A wrote:<br></div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">Hi Michael:<br><br> 1. After running a .swift or .dtm code, two files gets created: .xml
<div class="im"><br> and .klm. What do they represent?<br></div></blockquote><br>.xml is an xml version of the parsed .swift file<br>.kml (not klm) is the xml representation of the Karajan script that the Swift script is translated into for execution. Its actually the .kml file that is executed by Karajan which drives the execution of a Swift script.<br>
<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> 2. Correct me if I am wrong:
<div class="im"><br> * Datasets are mapped to physical presentation using mapping<br> algorithms. Some mapping algorithms already created part of<br> swift and the user can add/create others and use the<br>
existing once as the base.<br></div></blockquote><br>Yes, thats right.<br><br>But, to clarify this part:<br> "user can add/create others and use the existing ones as the base."<br><br>The user can use existing mappers, and add new mappers, either in Java or as external executables or scripts. But each mapper is independent. When you say "can use existing ones as a base" I would say thats correct, in that a user could *copy* and modify the code of one mapper to create another mapper, or, in the case of an "ext" mapper, one ext mapper could conceivably execute another and modify/filter its output to create a new mapping.<br>
<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> * Currently, the physical representation are files.<br></blockquote><br>Yes, if you mean to say that mappers map files to Swift variables.<br>
<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> 3. In the fMRI example, I see volume, Image, etc declared as
<div class="im"><br> a type? who defines them as a type?<br></div></blockquote><br><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> 4. In one of your emails, you stated that Swift functions can take
<div class="im"><br> accept files, int, string, float and boolean values as arguments.<br> They return files, or scalar values inside files. My question is: if the output is a string that is inside a file, how can I use<br>
this output in another program that takes it as an input? doesn't<br> call the file name and should have a code to read from the file?<br></div></blockquote><br>Yes, you can use readData() or readData2() to read the contents of a file back into Swift variables (including into arrays and structures, if the output has some structure).<br>
<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> 5. I am still confused when talk about XML Data Type and Mapping. Where is the XML representation? Is it the .xml that gets
<div class="im"><br> generated when run the swift code?<br></div></blockquote><br>No, the XML - if indeed it still exists - is only internal. I described it this way in an earlier post:<br><br>--
<div class="im"><br><br>"As Swift evolved from its early prototypes to a more mature system, the notion of XDTM evolved to one of mapping between filesystem-based structures and Swift in-memory data structures (ie, scalars, arrays, and structures, which can be nested and typed).<br>
<br></div>This is best seen by looking at the "external" mapper, ...
<div class="im"><br><br>In other words, it still has the flavor of XDTM, but without any XML being visible to the user. It meets the same need but is easier to use and explain."<br><br></div>--<br><br>When XDTM was first implemented, by Yong Zhao, he used XML within Swift to represent the mapping. I am not even sure if this XML representation is still used in the current implementation, or not. I suspect *not*.<br>
<br>But the important concept here should really be called "DTM" - dataset typing and mapping - and its embodied in the type model and mapping model of the language.<br><br>So you should stop thinking about data typing and mapping as being connected in any way to XML.<br>
<br>What we described in earlier papers as XDTM is not something that you can experiment with in terms of XML: ie, you can not see the XML for a mapping because its either deep inside the Swift implementation, or it no longer exists in the current Swift code.<br>
<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> 6. Let's look at this example:
<div class="im"><br><br> type messagefile {}<br> (messagefile t) greeting (string s[]) { app {<br> echo s[0] s[1] s[2] stdout=@filename(t<br></div> <mailto:<a href="mailto:stdout" target="_blank">stdout</a>=@filename(t>);
<div class="im"><br> }<br> }<br> messagefile outfile <"q5out.txt">;<br> string words[] = ["how","are","you"];<br> outfile = greeting(words);<br> ===<br>
So we have messagefile as a data type. outfile and words are<br> datasets. what will be the physical representation for these 2<br> datasets?<br></div></blockquote><br>An object of type messagefile will be represented as a single physical file externally, and internally as a scalar variable.<br>
<br>Words is a an array of strings.<br><br>Each atomic Swift variable (ie, scalars, array members, and structure members) can be thought of as a triple:<br><br> (set-state, mapping, value)<br><br>All variables have a set-state; initially unset, then set when the variable is assigned a value.<br>
<br>File-valued variables have only a mapping, but no value.<br>Scalar-values (ie, non-mapped variables like strings, as in your example) have a value (eg the string, interger, boolean or float value) but no mapping.<br><br>
We're still looking for better terminology to describe this; the current user guide uses both the terms "mapped type" and "marker type" to denote a file-valued variable. Both terms refer to the same concept; Im leaning to the term "mapped type".
<div class="im"><br><br>is thee system parsing the swift code, identifying the<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> data types and datasets and based on that choosea the proper mapping<br> algorithm needed?<br></blockquote>
<br></div>After the Swift command parses the Swift code, execution begins - i.e. the .kml file is executed by Karajan. Mappers are called as can be seen the kml. (And you can see their actions in the swift .log file).<br>
<br>The mapping for all mapped variables is either specified by the user (the most common case) or defaults to concurrent_mapper.<br><br>The users guide describes this in pretty good detail.<br><br>I hope that gets you a bit further. I hope that looking at XML mappings is not critical to your research, as I don't think you'll be able to readily get an XML intermediate form out of Swift.<br>
<br>An interesting topic would be to implement mechanisms to handle data in XML representations, in particular to enable Swift to invoke SOAP services as well as file-based applications and to compose scripts that call both forms of application.<br>
<br>- Mike<br><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> <br>Thanks,<br>Jamal
<div>
<div></div>
<div class="h5"><br> <br> <br><br><br>On Sun, Jul 26, 2009 at 9:53 PM, Michael Wilde <<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a> <mailto:<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>>> wrote:<br>
<br> Hi Jamal,<br><br> A lot of this is covered in the Swift user guide and tutorial. Have<br> you read through those yet?<br><br> All the docs are at: <a href="http://www.ci.uchicago.edu/swift/docs/index.php" target="_blank">http://www.ci.uchicago.edu/swift/docs/index.php</a><br>
<br> If so, and the clarifications below don't help, please ask again on<br> the list, OK?<br><br> - Mike<br><br><br><br> On 7/26/09 7:27 PM, J A wrote:<br><br> Hi Michael:<br> First, thank you for your reply and information provided.<br>
I am trying to understand more how it handles the input/output<br> parameters and make them available for other functions.<br><br><br> All functions in Swift are either atomic interfaces to application<br>
programs (ie, how o exec the program) or composite higher level<br> functions.<br><br> To illustrate, I will give this example for the sake of discussion:<br> I have a C program called test.c that contains 4 functions (<br>
main(), F1, F2, and F3). each function takes some parameters<br> such as int, string, name of a file that is in the same<br> directory, and each one produced some output (string, int, and a<br> file). Of course i am using global variables. Now, main calls<br>
F1, F1 passes its output to F2, and F2 passes its output to F3.<br><br><br> Swift doesnt look at the functions inside an application. It invokes<br> the application as a program (think fork/exec) just like a shell<br>
would, but distributed and in parallel if so specified.<br><br> Overall, the test.c takes an int, string, and file, and output<br> several files. the output files contains output produced by the<br> internal functions (tasks).<br>
<br><br> Swift functions can take accept files, int, string, float and<br> boolean values as arguments. They return files, or scalar values<br> inside files. (Again, think shell scripts). Composite structures -<br>
structs and arrays - of the above can be passed.<br><br> I would like to understand more when i transfer my code to<br> Swift how it handles the input/output data, where it stores<br> them, etc. I read couple of papers about XDTM and still have<br>
some confusion about the terms: dataset, typed, how/where its<br> physical representation is located at, and how the input/output<br> is used within the internal functions.<br><br><br> Files are by default named ("mapped") relative to the directory in<br>
which you run the Swift command. Many flexible extensions to that<br> model are provided for (eg, URIs). Swift sends the data to the site<br> chosen for execution (thats yet another topic) and returns results<br> back to the same submission host.<br>
<br> Mapping declarations in the Swift script specify how files and<br> directory structures are mapped to Swift variables (scalars, arrays,<br> structures). These are used in the specification of the Swift code.<br>
When Swift runs programs, it takes files that were mapped and knows<br> how to send them to grid sites or clusters and get data back.<br><br> I am new to this area and trying to understand how the DTM works.<br>
Any help from your side on this area is really appreciated.<br> Thanks,<br> Jamal<br> On Sun, Jul 26, 2009 at 7:09 PM, Michael Wilde<br> <<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a> <mailto:<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>><br>
</div></div>
<div>
<div></div>
<div class="h5"> <mailto:<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a> <mailto:<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>>>> wrote:<br><br> Jamal,<br>
<br> As Swift evolved from its early prototypes to a more mature<br> system,<br> the notion of XDTM evolved to one of mapping between<br> filesystem-based structures and Swift in-memory data<br>
structures (ie,<br> scalars, arrays, and structures, which can be nested and typed).<br><br> This is best seen by looking at the "external" mapper, which<br> allows<br> a user to map a dataset using any external program (typically a<br>
script) that returns the members of the dataset as a two-column<br> list: the Swift variable reference, and the external file or URI.<br><br> See the user guide section on the external mapper:<br>
<br> <a href="http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.ext_mapper" target="_blank">http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.ext_mapper</a><br> (but the example in the user guide doesn't show the power of<br>
mapping<br> to nested structures).<br><br> In other words, it still has the flavor of XDTM, but without<br> any XML<br> being visible to the user. It meets the same need but is<br> easier to<br>
use and explain.<br><br> - Mike<br><br><br> On 7/26/09 2:50 PM, J A wrote:<br><br> Hi All:<br> Can any one direct me to a source with more<br> examples/explanation on how XDTM is working/implemented?<br>
Thanks,<br> Jamal<br> ------------------------------------------------------------------------<br><br> _______________________________________________<br>
Swift-user mailing list<br> <a href="mailto:Swift-user@ci.uchicago.edu" target="_blank">Swift-user@ci.uchicago.edu</a><br> <mailto:<a href="mailto:Swift-user@ci.uchicago.edu" target="_blank">Swift-user@ci.uchicago.edu</a>><br>
<mailto:<a href="mailto:Swift-user@ci.uchicago.edu" target="_blank">Swift-user@ci.uchicago.edu</a><br> <mailto:<a href="mailto:Swift-user@ci.uchicago.edu" target="_blank">Swift-user@ci.uchicago.edu</a>>><br>
<br> <a href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-user" target="_blank">http://mail.ci.uchicago.edu/mailman/listinfo/swift-user</a><br><br><br><br></div></div></blockquote></blockquote></div><br>