[Swift-user] running swift k on multiple instances on EC2, using coaster-service

Wed Oct 24 12:13:06 CDT 2012

That sounds great! Iman, can you give Ketan's image a try, and report back if it worked, or any other problems encountered? 

Ioan

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor
=================================================================
Computer Science Department
Illinois Institute of Technology
10 W. 31st Street Chicago, IL 60616 =================================================================
Cel:   1-847-722-0876
Email: iraicu at cs.iit.edu
Web:   http://www.cs.iit.edu/~iraicu/
=================================================================
=================================================================

On Oct 24, 2012, at 12:11 PM, Ketan Maheshwari <ketancmaheshwari at gmail.com> wrote:

> Hi Ioan,
> 
> I've been extensively using Swift on Amazon EC2 via coasters. 
> 
> I am using a fedora 8 image. 
> 
> I use an external host as a submit host so, only perl (for worker.pl; is installed) and application binaries needs to be on EC2 nodes.
> 
> The name of my image is ketan_mars and is on us-east-1 region.
> 
> Let me know if you have more questions in this regard. 
> 
> Regards,
> Ketan
> 
> On Wed, Oct 24, 2012 at 12:47 PM, Ioan Raicu <iraicu at cs.iit.edu> wrote:
>> Hi David,
>> Has anyone in the Swift group run Swift on Amazon? Perhaps there are
>> already some pre-configured images Iman can use, that already has the
>> environment (Linux, Java, Swift, PBS/Condor, GridFTP, NFS/PVFS, etc)
>> setup and ready to use. If there are such images, please point Iman in
>> the right direction. If not, perhaps after Iman finishes his
>> configuration, we can reference his images for others to use on Amazon.
>> Ioan
>> 
>> On 10/24/2012 10:28 AM, David Kelly wrote:
>> > Iman,
>> >
>> > There is some documentation on how to use the start-coaster-service script at http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html#_bag_of_workstations. You should be able to follow the "bag of workstations" configuration example once you have started your EC2 instance. That may help to simplify things for you a bit. Please let me know if you have any questions or issues with this.
>> >
>> > Thanks,
>> > David
>> >
>> > ----- Original Message -----
>> >> From: "Iman Sadooghi" <isadoogh at iit.edu>
>> >> To: swift-user at ci.uchicago.edu
>> >> Sent: Tuesday, October 23, 2012 5:46:57 PM
>> >> Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service
>> >> Hi everyone
>> >>
>> >>
>> >> I am trying to run a Montage application workflow with swift on
>> >> multiple instances of AMAZON EC2.
>> >> So far I was able to set up a cluster, and a PVFS files system shared
>> >> among the nodes ( using FUSE. so I will have POSIX interface on my
>> >> swift work directory ).
>> >> I have tried running a simple hello.swift example on multiple nodes
>> >> with the coaster. the working directory is the shared folder
>> >> (supported by PVFS).
>> >> when I run the code using my own tc.data and sites.xml, this will
>> >> happen:
>> >>
>> >>
>> >>
>> >> (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data
>> >> -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift
>> >> (results:)
>> >> Swift 0.93 swift-r5483 cog-r3339
>> >>
>> >>
>> >> RunID: 20121023-2200-4d3knr72
>> >> Progress: time: Tue, 23 Oct 2012 22:00:50 +0000
>> >> Find: http://10.244.4.101:1213
>> >> Find: keepalive(120), reconnect - http://10.244.4.101:1213
>> >> Passive queue processor initialized. Callback URI is
>> >> http://10.244.4.101:1212
>> >> Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1
>> >> Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1
>> >> Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1
>> >> Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1
>> >> Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1
>> >> Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1
>> >> Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1
>> >>
>> >>
>> >> and it keeps doing this forever meaning that there is no answer from
>> >> worker nodes!
>> >> as I checked on worker nodes, the working files are created on the
>> >> shared folder, and when i check the running applications, there is a
>> >> java application running. but nothing happens.
>> >> I have also attached the log file of my hello.swift running in case
>> >> you need to take a look at it.
>> >> should I consider using pbs, or condor,... I have no idea about how
>> >> they work though.
>> >>
>> >>
>> >> I appreciate if anyone can help me with it. Thank you so much.
>> >>
>> >> Best, --
>> >> Iman Sadooghi
>> >> Illinois Institute of Technology (IIT)
>> >> Data-Intensive Distributed Systems Laboratory
>> >>
>> >>
>> >> _______________________________________________
>> >> Swift-user mailing list
>> >> Swift-user at ci.uchicago.edu
>> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> > _______________________________________________
>> > Swift-user mailing list
>> > Swift-user at ci.uchicago.edu
>> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> 
>> --
>> =================================================================
>> Ioan Raicu, Ph.D.
>> Assistant Professor, Illinois Institute of Technology (IIT)
>> Guest Research Faculty, Argonne National Laboratory (ANL)
>> =================================================================
>> Data-Intensive Distributed Systems Laboratory, CS/IIT
>> Distributed Systems Laboratory, MCS/ANL
>> =================================================================
>> Cel:    1-847-722-0876
>> Office: 1-312-567-5704
>> Email:  iraicu at cs.iit.edu
>> Web:    http://www.cs.iit.edu/~iraicu/
>> Web:    http://datasys.cs.iit.edu/
>> =================================================================
>> =================================================================
>> 
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> 
> 
> -- 
> Ketan
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20121024/5ba9f666/attachment.html>