[Swift-user] Virtual data schema / catalog?

[Ben Clifford]

On Fri, 14 Sep 2007, Allen, M. David wrote:

> In other words, right now I can use swift to chain a bunch of programs
> together, but what if I want to take a work product, assign it to a
> human to do something with it, and then use the output of the human's
> effort to act as the input to the next step?  One way might be to
> intentionally stop the workflow at the point where the human should
> take the input.  (Alternately, to let it fail because the human work
> product doesn't exist yet)  The human would then go off and do whatever
> they're supposed to do, and subsequently upload the result to some
> website.  The act of uploading that result would then restart the
> workflow to continue processing with the human's result as an
> intermediate work product.  Viola.  Humans can be tasked to do
> arbitrarily complex things that computers can't do, just like they were
> an invocation of "grep".  :)

I think what you describe there (at least the swift side of it) is pretty 
much doable now. Make your application be:

  #!/bin/bash
  
  if [ -f /tmp/userresponse$1 ]; then
   cp /tmp/userresponse$1 $2
   exit 0;
  else
   exit 1;
  fi

And have the first parameter be a string (not a mapped filename) that 
describes or identifies the request; and $2 be a mapped filename that will 
give the output when its ready.


Another way you could implement that is write a custom execution provider 
(i.e. the abstraction that lets swift submit a job to run locally using 
the fork provider, through Globus using the globus provider, through PBS 
using the PBS provider). You could write a provider-human. In that model, 
Swift would keep running, treating your human job the same as any other 
long running queued job; that may or may not be desirable.

> On a separate note, (the XML schema) - I will be using some of the
> nightly builds.  Is the XML schema fairly stable?  How often does it
> change?

There was a massive rewrite that went in a week or so ago. The previous 
change before that was in Febuary and before that was the import into our 
SVN repository in December 2006. So historically its been fairly stable.

Where its likely to change is when new language features are added - for 
example, I have a patch that provides a different kind of iteration loop. 
That adds a new language construct to the SwiftScript text syntax and a 
corresponding XML representation.

--