[Swift-devel] Swift-issues (PBS+NFS Cluster)

[Ben Clifford]

On Thu, 7 May 2009, Michael Wilde wrote:

> I wonder if the Swift engine could pull the data back from a GridFTP server
> running on the worker node?

There's nothing virtual-machine specific here, I think.

>From the perspective of the cio work:

If the site shared filesystem is some distributed fs (which is what I 
think the collective I/O stuff mostly is) then a CIO-aware gridftp client 
could choose one of several gridftp servers that have access to the same 
cio file system, and select a preferable one to contact.

That could fit into Swift at the cog file transfer layer, as an 
alternative provider to gridftp.

It would preserve Swift's view of sites as fairly monolithic entities, 
with intra-site selection activity made by lower levels in the stack (as 
happens with pbs or falkon or coasters).

That hierarchy feels more scalable to me - it works ok in the job 
submission scenario with worker nodes being allocated by LRMs, for 
example.

--