[Swift-devel] Interesting observation when running Swift

[Mihael Hategan]

That's the VDC scenario to a large extent, where the VDC is a filesystem
with metadata and a few nice things like locking, replication, and
listeners (publish/subscribe). In principle, this is what the futures
+Swift types are: an in-memory file system with a p/s mechanism.

Efficient implementation of such things on top of existing filesystems
is, I think, not that easy. Note that the p/s mechanism would need to be
emulated (perhaps by polling the disks). Location-independence may also
prove tricky, since you'd need to support these for things like GridFTP.

So yes, it would be useful, but pretty difficult, and then you start
asking yourself if another solution wouldn't achieve the same thing
faster.

On Tue, 2007-04-10 at 11:57 -0500, Tiberiu Stef-Praun wrote:
> Interesting.
> 
> Does anyone else think that monitoring the filesystem could be a useful idea ?
> 
> For instance it could help with file-driven dependencies, in scenarios
> where we want to have continuous workflows, or compose independent
> wokflows. The filesystem would act as the publish-subscribe mechanism
> for some workflow cases.
> 
> Tibi
> 
> On 4/10/07, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > Swift doesn't monitor the file system.
> > Data driven doesn't mean that it does magic in the background. It means
> > that you have to express data dependencies in the code.
> >
> > On Tue, 2007-04-10 at 11:47 -0500, Tiberiu Stef-Praun wrote:
> > > I have a workflow along these lines:
> > >
> > > // this one generates outputsStage1[]
> > > stage1()
> > > // this one merges the stage1 outputs
> > > stage2(outputsStage1[])
> > >
> > > note that it is not outputsStage1=stage1()
> > >
> > > Since the outputsStage1 files were not generated yet, I expected that
> > > Karajan waited for them to be created before running stage2, but that
> > > was not the case: stage2 was executed when the workflow started (and
> > > it failed) and caused the workflow to fail.
> > >
> > > I know how to fix the workflow, that is not the issue. The issue is
> > > that I expected the workflow to be data-driven, but it seems to be
> > > code driven. Explanation: it attempted to execute a section even if
> > > its input files were not available.
> > >
> > > Correct me if I am wrong.
> > > Tibi
> > >
> >
> >
> 
>