[Swift-devel] a couple of file staging management ideas in the next 3 months

[Mihael Hategan]

I'm a bit fuzzy on the kind of generic changes that would facilitate the
move to Falkon. I'd rather like to see the concrete things that are
needed.

On Tue, 2008-12-02 at 19:55 +0000, Ben Clifford wrote:
> Here are a couple of ideas that might be useful to implement over the next 
> three months.
> 
> This is intended to be read in context of two previous notes documenting 
> what I think is the abstract model that SwiftScript provides, and the 
> present Swift implementation of that.
> 
> These notes refer to changes that could reasonably be made to the 
> production Swift in the next ~3 months, to both facilitate the 
> experimental side of things (i.e. Falkon) and to broaden the range of 
> sites which Swift could be used on in production (i.e. gLite).
> 
> Specifically, they are not intended discuss approaches to architecting 
> Falkon, other than to discuss how approaches in Falkon might be coherently 
> interfaced to Swift.
> 
> Two distinct approaches (which are complimentary and could both 
> implemented) seem feasible to implement in the next two months and 
> interesting at the Swift layer (in as much as they facilitate ongoing 
> work):
> 
>  * change wrapper.sh to access the shared filesystem in various
>    selectable/configurable/pluggable ways; keep everything else the same.
> 
>    So, this changes the requirement that wrapper.sh have posix access to
>    the site cache into a requirement that there is command-line scriptable
>    access to the site cache.
> 
>    This would facilitate gLite's model of no posix access to the site
>    storage system without radically changing architecture. Some OSG users 
>    have suggested that this is desirable on OSG.
> 
>  * Fiddle with the abstraction at the execute2 layer
> 
>    execute2 is a routine in the Swift library that selects a site,
>    and then causes stagein to site shared directory, execution on the
>    remote site, and stageout from that site to occur.
> 
>    This layer could be adjusted so the sequence: stagein/execute/stageout
>    is pluggable. By default the exist implementation would be used; but it
>    would allow other code to be plugged in that could replace both the
>    existing client site stagein/execute/stageout code and the remote
>    wrapper.sh code. This would facilitate running in environments where
>    the same underlying layer wants to do both data management and
>    execution management (such as falkon)
> 
>    In the falkon case, what is now provider-deef would plug in at this
>    level and offload all staging responsibility from core Swift.
>