[Swift-devel] Issues in worker-side GridFTP

[Michael Wilde]

We have been discussing the feasibility of adding a data management option to Swift that performs data transfer via globus-url-copy from the worker node, likely issued from within a CDM function.

This is mainly intended for OSG sites where no other straightforward transfer option exists (e.g. on sites that have storage servers like SRM-DCache and no mounted access to the primary GridFTP server's filesystem).

I'm wondering how worker-side GridFTP will work with respect to data management functions like creation of the work directory, job directory, and transfer of utility files like swiftwrap etc.

Any ideas on how best to do this? Its almost like we should first try this in coasters using provider staging, where we start by replacing the data transfer with a worker-side invocation of guc.

Mihael, is this something that you could do for us?  If you could create a test version of this using a simple "cp" in a worker.pl callout script for doing data transfer, we could then do the testing with actual guc.

- Mike