[Swift-user] XDTM

[Michael Wilde]

Hi Jamal,

A lot of this is covered in the Swift user guide and tutorial. Have you 
read through those yet?

All the docs are at: http://www.ci.uchicago.edu/swift/docs/index.php

If so, and the clarifications below don't help, please ask again on the 
list, OK?

- Mike


On 7/26/09 7:27 PM, J A wrote:
> Hi Michael:
>  
> First, thank you for your reply and information provided.
>  
> I am trying to understand more how it handles the input/output 
> parameters and make them available for other functions.

All functions in Swift are either atomic interfaces to application 
programs (ie, how o exec the program) or composite higher level functions.
>  
> To illustrate, I will give this example for the sake of discussion:
>  
> I have a C program called test.c that contains 4 functions ( main(), F1, 
> F2, and F3).  each function takes some parameters such as int, string, 
> name of a file that is in the same directory, and each one produced some 
> output (string, int, and a file).  Of course i am using global 
> variables.  Now, main calls F1, F1 passes its output to F2, and F2 
> passes its output to F3.

Swift doesnt look at the functions inside an application. It invokes the 
application as a program (think fork/exec) just like a shell would, but 
distributed and in parallel if so specified.
>  
> Overall, the test.c takes an int, string, and file, and output several 
> files.  the output files contains output produced by the internal 
> functions (tasks).

Swift functions can take accept files, int, string, float and boolean 
values as arguments. They return files, or scalar values inside files. 
(Again, think shell scripts).  Composite structures - structs and arrays 
- of the above can be passed.
>  
> I would like to understand more when i transfer my code to Swift how it 
> handles the input/output data, where it stores them, etc.  I read couple 
> of papers about XDTM and still have some confusion about the terms:  
> dataset, typed, how/where its physical representation is located at, and 
> how the input/output is used within the internal functions.

Files are by default named ("mapped") relative to the directory in which 
you run the Swift command. Many flexible extensions to that model are 
provided for (eg, URIs).  Swift sends the data to the site chosen for 
execution (thats yet another topic) and returns results back to the same 
submission host.

Mapping declarations in the Swift script specify how files and directory 
structures are mapped to Swift variables (scalars, arrays, structures). 
These are used in the specification of the Swift code. When Swift runs 
programs, it takes files that were mapped and knows how to send them to 
grid sites or clusters and get data back.
>  
>  
> I am new to this area and trying to understand how the DTM works.
>  
> Any help from your side on this area is really appreciated.
>  
> Thanks,
> Jamal
>  
> 
>  
> On Sun, Jul 26, 2009 at 7:09 PM, Michael Wilde <wilde at mcs.anl.gov 
> <mailto:wilde at mcs.anl.gov>> wrote:
> 
>     Jamal,
> 
>     As Swift evolved from its early prototypes to a more mature system,
>     the notion of XDTM evolved to one of mapping between
>     filesystem-based structures and Swift in-memory data structures (ie,
>     scalars, arrays, and structures, which can be nested and typed).
> 
>     This is best seen by looking at the "external" mapper, which allows
>     a user to map a dataset using any external program (typically a
>     script) that returns the members of the dataset as a two-column
>     list: the Swift variable reference, and the external file or URI.
> 
>     See the user guide section on the external mapper:
> 
>      http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.ext_mapper
>     (but the example in the user guide doesn't show the power of mapping
>     to nested structures).
> 
>     In other words, it still has the flavor of XDTM, but without any XML
>     being visible to the user. It meets the same need but is easier to
>     use and explain.
> 
>     - Mike
> 
> 
>     On 7/26/09 2:50 PM, J A wrote:
> 
>         Hi All:
>          Can any one direct me to a source with more
>         examples/explanation on how XDTM is working/implemented?
>          Thanks,
>         Jamal
>          
> 
>         ------------------------------------------------------------------------
> 
>         _______________________________________________
>         Swift-user mailing list
>         Swift-user at ci.uchicago.edu <mailto:Swift-user at ci.uchicago.edu>
>         http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 
>