[Swift-devel] Swift-issues (PBS+NFS Cluster)

[Ioan Raicu]

Hi,

yizhu wrote:
> Hi,
>
> I've now got the swift system running on Amazon EC2 with swift 
> installed on head-node of the cluster. I will soon let it the user 
> submit the job from its own  machine after i solve the authentication 
> issues of Globus.
>
>
> I think my next step is to write a sample swift code to check if we 
> can let swift grab input files from S3, execute it, and then write the 
> output files back to S3.
>
>
> Since each Amazon virtual node only has limited storage space( Small 
> instance for 1.7GB, Large Instance for 7.5GB), 
Are you sure? That sounds like like the amount of RAM each instance 
gets. The last time I used EC2 (more than a year ago), each instance had 
a disk of 100GB+ each, which could be treated as a scratch disk that 
would be lost when the instance was powered down.
> we may need to use EBS(Elastic Block Store) to storage temp files 
> created by swift. The EBS behaved like a hard disk volume and can be 
> mounted by any virtual nodes. But here arise a problem, since a volume 
> can only be attached to one instance at a time[1], so the files store 
> in EBS mounted by one node won't be shared by any other nodes; 
The EBS sounds like the same thing as the local disk, except that it is 
persisted in S3, and can be recovered when the instance is started again 
later.

You can use these local disks, or EBS disks, to create a shared/parallel 
file system, or manage them yourself.
> we lost the file sharing ability which I think is a fundamental 
> requirement for swift.
The last time I worked with the Workspace Service (now part of Nimbus), 
we were able to use NFS to create a shared file system across our 
virtual cluster. This allowed us to run Swift without modifications on 
our virtual cluster. Some of the more recent work on Swift might have 
eased up some of the requirements for a shared file system, so you might 
be able to run Swift without a shared file system, if you configure 
Swift just right. Mike, is this true, or we aren't quite there yet?

Ioan
>
>
> -Yi
>
>
>
>
>
> Michael Wilde wrote:
>>
>>
>> On 5/7/09 12:54 PM, Tim Freeman wrote:
>>> On Thu, 07 May 2009 11:39:40 -0500
>>> Yi Zhu <yizhu at cs.uchicago.edu> wrote:
>>>
>>>> Michael Wilde wrote:
>>>>> Very good!
>>>>>
>>>>> Now, what kind of tests can you do next?
>>>> Next, I will try to let swift running on Amazon EC2.
>>>>
>>>>> Can you exercise the cluster with an interesting workflow?
>>>> Yes, Is there any complex sample/tools i can use (rahter than 
>>>> first.swift) to test swift performance? Is there any benchmark 
>>>> available i can compare with?
>>>>
>>>>> How large of a cluster can you assemble in a Nimbus workspace ?
>>>> Since the vm-image i use to test 'swift' is based on NFS shared 
>>>> file system, the performance may not be satisfiable if the we have 
>>>> a large scale of cluster. After I got the swift running on Amazon 
>>>> EC2, I will try to make a dedicate vm-image by using GPFS or any 
>>>> other shared file system you recommended.
>>>>
>>>>
>>>>> Can you aggregate VM's from a few different physical clusters into 
>>>>> one Nimbus workspace?
>>>> I don't think so. Tim may make commit on it.
>>>
>>> There is some work going on right now making auto-configuration 
>>> easier to do
>>> over multiple clusters (that is possible now, it's just very 
>>> 'manual' and
>>> non-ideal unlike with one physical cluster).  You wouldn't really 
>>> want to do NFS
>>> across a WAN, though.
>>
>> Indeed. Now that I think this through more clearly, one workspace == 
>> one cluster == one Swift "site", so we could aggregate the resources 
>> of multiple workspaces through Swift to execute a multi-site workflow.
>>
>> - Mike
>>
>>>>
>>>>> What's the largest cluster you can assemble with Nimbus?
>>>> I am not quite sure,I will do some test onto it soon. since it is a 
>>>> EC2-like cloud, it should easily be configured as a cluster with 
>>>> hundreds of nodes. Tim may make commit on it.
>>>
>>> I've heard of EC2 deployments in the 1000s at once, it's up to your 
>>> EC2 account
>>> limitations (they seem pretty efficient with making your quota 
>>> ever-higher).
>>> Nimbus installation at Teraport maxes out at 16, there are other 
>>> 'science
>>> clouds' but I don't know their node numbers.  EC2 is the place where 
>>> you will
>>> be able to really test scaling something.
>>>
>>> Tim
>>
>
>

-- 
===================================================
Ioan Raicu, Ph.D.
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
http://dev.globus.org/wiki/Incubator/Falkon
http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page
===================================================
===================================================