[Swift-devel] Swift-issues (PBS+NFS Cluster)

[Tim Freeman]

On Wed, 13 May 2009 16:58:50 +0000 (GMT)
Ben Clifford <benc at hawaga.org.uk> wrote:

> 
> OK.
> 
> Specifically for that problem, then, something that makes the S3 storage 
> available at the swift submit side would avoid transferring data across 
> the $-boundary and would avoid having to mess too much with the data 
> management architecture.
> 
> That could be either a cog provider to pull files from s3 (the existing 
> dcache provider might serve as an example - it seems conceptually quite 
> similar); or a posix mount of s3 on the submit node (in which case the 
> local filesystem cog provider would work).

This package looks good and is apache2 licensed:

http://code.google.com/p/jclouds/

Tim

> 
> This would not transfer data from s3 directly to the worker nodes, but 
> that is not the problem being addressed here.
> 
> On Wed, 13 May 2009, foster at anl.gov wrote:
> 
> > The reason for wanting to use S3 is that one pays to move data to/from
> > amazon. So, we want to allow analysis of data that sits on S3, and puts
> > it's output on s3
> > 
> > 
> > Ian -- from mobile
> > 
> > On May 13, 2009, at 12:25 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
> > 
> > > 
> > > On Wed, 13 May 2009, Ben Clifford wrote:
> > > 
> > > > > Since each Amazon virtual node only has limited storage space( Small
> > > > > instance
> > > > > for 1.7GB, Large Instance for 7.5GB), we may need to use EBS(Elastic
> > > > > Block
> > > > > Store) to storage temp files created by swift. The EBS behaved like a
> > > > > hard
> > > 
> > > For many applications, 1.7G is a very large amount of data.
> > > 
> > > 7.5G of storage for your shared filesystem and 1.7G for worker node
> > > temporary files seems like a lot to me.
> > > 
> > > So I think its important to clarify: do you want to use s3 because you
> > > think it solves a storage problem, and if so, does that problem really
> > > exist now? or do you want to use s3 because it is an interesting thing to
> > > play with? (either way is fine, but I think it is important to clarify)
> > > 
> > > -- 
> > > 
> >