[Swift-devel] Streams

[Michael Wilde]

Maybe we could come close with an (~infinite) server loop, and an ext mapper invocation in that loop that pulled data from some IPC interface and created files for Swift to process on each trip around the loop.

Thats essentially what the SwiftR service does, using FIFOs to do it all with no code changes to Swift itself (for the moment, just to play with the mechanism and paradigm).

I think you could experiment with this approach as a frst approximation, Ketan.

Getting and adapting the SwiftR code *might* be a start. In fact you could simulate the input stream by creating it or reading it from an R driver program (which can do socket calls).

- Mike


----- Original Message -----
> From: "Tim Armstrong" <tim.g.armstrong at gmail.com>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>, "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Tuesday, May 1, 2012 1:52:01 PM
> Subject: Re: [Swift-devel] Streams
> My instinct is that if you wanted to do this in Swift, then the right
> way would be to model a stream as a variable in the same way as files.
> Swift would be responsible for hooking up applications to different
> streams... You can't do this in current Swift, I'm guessing it would
> need major extensions
> 
> - Tim
> 
> 
> On Tue, May 1, 2012 at 12:48 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> Hi Ketan,
> 
> I'll try to reply to this and Ben's followup some time next week. Im
> very interested in the streams paradigm.
> 
> Note that we have at least one example of a Swift script running in an
> infinite loop: its the Swift remote function call server in the SwiftR
> package (Swift for the R language). I think that uses an infinite
> iterate loop that exits when it gets an exit command from the FIFO its
> reading. We talked about adding simple socket interfaces via builtin
> functions.
> 
> The streams question is of string interest for the future.
> 
> - Mike
> 
> 
> 
> 
> ----- Original Message -----
> > From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
> > To: "Swift Devel" < swift-devel at ci.uchicago.edu >
> > Sent: Monday, April 30, 2012 1:39:54 PM
> > Subject: [Swift-devel] Streams
> > Hi,
> >
> >
> > I am working on a DOE powergrid related project here at Cornell.
> >
> >
> > An aim of the project is to compute power grid state estimation and
> > react in time-critical fashion.
> >
> >
> > The application at a very high level, is a distributed
> > producer/consumer system where multiple producers produce data
> > streams
> > consumed by multiple consumers in a publish-subscribe model of data
> > flow.
> >
> >
> > The producers (phasor measurement units) produce streams
> > continuously
> > and consumers(State Estimators) can subscribe to the producers.
> > There
> > can be multiple consumers consuming from a single producer for
> > performance and consistency purposes.
> >
> >
> > Can Swift support this model of computation? In particular, I am
> > wondering how to go about the following aspects with Swift:
> >
> >
> > 1. Describe application which could run in an 'infinite' loop.
> > 2. Mappers to streams. I think these streams should be some kind of
> > named buffers. A memory to memory stream model is what I vaguely
> > view
> > this as.
> >
> >
> > The streams are binary encoded ones in big-endian format and could
> > be
> > parsed (by consumers) as id'd tuples each containing 5-6 fixed width
> > field of timestamp, voltage, current, delta etc. data.
> >
> >
> > There are other requirements of the application and plenty of low
> > level nitty gritty but I think Swift could handle all of'em. I am
> > just
> > unsure of the above two at the moment.
> >
> >
> > We are in discussion with WSU collaborators to deliver some of the
> > 'real' parts of the application. However, in the meantime we do have
> > toy components to test and play with.
> >
> >
> > Any input is greatly appreciated.
> >
> >
> > Regards,
> > --
> > Ketan
> >
> >
> >
> 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory