[Swift-devel] provider staging stuff

[Mihael Hategan]

This email is to mention a few ideas regarding the issue of
site-selectable provider staging.

Currently, provider staging can only be enabled globally. Somewhere in
swift.k there is a conditional import of either swift-int.k or
swift-int.k (or the more recent swift-int-wrapperstaging.k). The nice
thing about the global switch is that it is evaluated at karajan-compile
time, and the import and compilation of the relevant swift-int*.k is
done before anything is executed.

I want to replace that by:
1. eliminating all swift-int*.k files
2. moving that logic into Java code

Swift needs staging, that's clear. However, it doesn't care how it's
done as long as the files are there when the job is started and they are
moved back when the job is done.

Some providers support staging. There are two levels of support for
this. One is the basic level that GRAM does, and the other is the
improved version that coasters and the local provider do, which support
conditional staging based on job status and lenient staging (i.e. stage
a file only if it's there). Of course, some providers don't support
staging at all.

However, this is an implementation level issue. Out of a non-staging
provider and a file op provider, one can build a combined provider that
does support file staging. This is a pretty old idea, before swift, that
never got implemented because... I don't remember.

I think that doing this would be a good move since it would probably
simplify some swift code (I always thought swift-int.k was convenient
for prototyping, but ugly), would probably reduce memory consumption,
would make some of the logic useable outside of swift/k and, of course,
enable finer control over how staging is done.

Mihael