[Swift-devel] Swift-issues (PBS+NFS Cluster)

[Ben Clifford]

OK.

Specifically for that problem, then, something that makes the S3 storage 
available at the swift submit side would avoid transferring data across 
the $-boundary and would avoid having to mess too much with the data 
management architecture.

That could be either a cog provider to pull files from s3 (the existing 
dcache provider might serve as an example - it seems conceptually quite 
similar); or a posix mount of s3 on the submit node (in which case the 
local filesystem cog provider would work).

This would not transfer data from s3 directly to the worker nodes, but 
that is not the problem being addressed here.

On Wed, 13 May 2009, foster at anl.gov wrote:

> The reason for wanting to use S3 is that one pays to move data to/from amazon.
> So, we want to allow analysis of data that sits on S3, and puts it's output on
> s3
> 
> 
> Ian -- from mobile
> 
> On May 13, 2009, at 12:25 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
> 
> > 
> > On Wed, 13 May 2009, Ben Clifford wrote:
> > 
> > > > Since each Amazon virtual node only has limited storage space( Small
> > > > instance
> > > > for 1.7GB, Large Instance for 7.5GB), we may need to use EBS(Elastic
> > > > Block
> > > > Store) to storage temp files created by swift. The EBS behaved like a
> > > > hard
> > 
> > For many applications, 1.7G is a very large amount of data.
> > 
> > 7.5G of storage for your shared filesystem and 1.7G for worker node
> > temporary files seems like a lot to me.
> > 
> > So I think its important to clarify: do you want to use s3 because you
> > think it solves a storage problem, and if so, does that problem really
> > exist now? or do you want to use s3 because it is an interesting thing to
> > play with? (either way is fine, but I think it is important to clarify)
> > 
> > -- 
> > 
>