[Swift-devel] a couple of file staging management ideas in the next 3 months

[Ben Clifford]

Here are a couple of ideas that might be useful to implement over the next 
three months.

This is intended to be read in context of two previous notes documenting 
what I think is the abstract model that SwiftScript provides, and the 
present Swift implementation of that.

These notes refer to changes that could reasonably be made to the 
production Swift in the next ~3 months, to both facilitate the 
experimental side of things (i.e. Falkon) and to broaden the range of 
sites which Swift could be used on in production (i.e. gLite).

Specifically, they are not intended discuss approaches to architecting 
Falkon, other than to discuss how approaches in Falkon might be coherently 
interfaced to Swift.

Two distinct approaches (which are complimentary and could both 
implemented) seem feasible to implement in the next two months and 
interesting at the Swift layer (in as much as they facilitate ongoing 
work):

 * change wrapper.sh to access the shared filesystem in various
   selectable/configurable/pluggable ways; keep everything else the same.

   So, this changes the requirement that wrapper.sh have posix access to
   the site cache into a requirement that there is command-line scriptable
   access to the site cache.

   This would facilitate gLite's model of no posix access to the site
   storage system without radically changing architecture. Some OSG users 
   have suggested that this is desirable on OSG.

 * Fiddle with the abstraction at the execute2 layer

   execute2 is a routine in the Swift library that selects a site,
   and then causes stagein to site shared directory, execution on the
   remote site, and stageout from that site to occur.

   This layer could be adjusted so the sequence: stagein/execute/stageout
   is pluggable. By default the exist implementation would be used; but it
   would allow other code to be plugged in that could replace both the
   existing client site stagein/execute/stageout code and the remote
   wrapper.sh code. This would facilitate running in environments where
   the same underlying layer wants to do both data management and
   execution management (such as falkon)

   In the falkon case, what is now provider-deef would plug in at this
   level and offload all staging responsibility from core Swift.

--