[Swift-user] Swift Coaster Service & Docker Containers

[Yadu Nand]

Hi Matthew,

You can use the coaster-service directly with the following arguments :
coaster-service -p $SERVICEPORT -localport $WORKERPORT -nosec -passive &>
/var/log/coaster-service.logs

The workers running under docker should be connecting to the WORKERPORT and
the swift client would use the SERVICEPORT.
Another thing to remember is to make sure that you use the the
coaster-service and worker.pl from the same swift release.

The start-coaster-service command is a wrapper over coaster-service which
in addition to starting the coaster service, also
tries to ssh the worker.pl to the WORKER_HOSTS. In the situation that you
describe, I think it makes sense to have the workers
just connect back to the coaster-service which is listening on a known IP /
port.

Thanks,
Yadu




On Fri, Aug 29, 2014 at 8:34 PM, Michael Wilde <wilde at anl.gov> wrote:

>  Matthew,
>
> I would treat this case similar to the configuration you'd use for a set
> of virtual machines.
>
> Start one coaster service for each pool of identical docker containers
> that you want to run.
>
> For each pool of containers, run a Swift worker (worker.pl) in the
> container and have it connect back to the coaster service you designate to
> manage that pool.
>
> I'm assuming that from a docker contained you can connect out to any
> reachable IP address.
>
> Another approach is to treat the containers like a set of ad-hoc compute
> nodes, and ssh into them with automatic coasters using the ssh:local
> jobmanager setting.
>
> We'll try to test and publish a config for such cases. We'd welcome your
> help with that.
>
> - Mike
>
>
>
> On 8/29/14, 7:48 PM, Matthew Shaxted wrote:
>
>  Hi All,
>
>
>
> I’m trying to find a way to run Swift workflows on multi-host docker
> containers, and wondering if anybody has had success with this.
>
>
>
> When I start a docker container and define specific ports to open on the
> container, they mapped to random ports on the host machine. So for example,
> I can start a container with an ssh port open from a host “10.1.1.1”, and
> can then access this container across hosts with “ssh compute at 10.1.1.1 –p
> 49160”.
>
>
>
> Now I’m hoping to link these docker containers to Swift’s
> start-coaster-service. I think it would be possible and relatively easy if
> I can say, start the coasters on a series of IP addresses AND ports. So a
> host’s file perhaps would look something like below, and coasters would be
> started on the correct docker container:
>
>
>
> 10.1.1.1: 49160
>
> 10.1.1.2: 34155
>
> 10.1.1.2: 34156   …
>
>
>
> Does this make sense? Is it possible to start coasters by specifying an IP
> address and port number?
>
>
>
> Any thoughts are greatly appreciated.
>
>
>
> Thanks,
>
> Matthew
>
>
>
>
>
> MATTHEW SHAXTED
>
> SKIDMORE, OWINGS & MERRILL LLP
> 224 SOUTH MICHIGAN AVENUE
> CHICAGO, IL 60604
> T  (312) 360-4368
> MATTHEW.SHAXTED at SOM.COM
>
>
>
> [image: cid:image001.png at 01CF9071.6FB46030] <http://www.som.com/>
>
> The information contained in this communication may be confidential, is
> intended only for the use of the recipient(s) named above, and may be
> legally privileged. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution, or
> copying of this communication, or any of its contents, is strictly
> prohibited and may be unlawful. If you have received this communication in
> error, please return it to the sen­­­der immediately and delete the
> original message and any copy of it from your computer system. If you have
> any questions concerning this message, please contact the sender.
>
>
>
> [image:
> http://intranet.som.com/common/admin/file.cfm?f=%2Fresources%2Fcontent%2F5%2F0%2F4%2F4%2F6%2F4%2F0%2F3%2Fdocuments%2Fimagea560bf%2Egif%406e10073b%2E30854c37]
>
>
>
>
> _______________________________________________
> Swift-user mailing listSwift-user at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
>
> --
> Michael Wilde
> Mathematics and Computer Science          Computation Institute
> Argonne National Laboratory               The University of Chicago
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>



-- 
Yadu Nand B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140829/3fbd300b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 566 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140829/3fbd300b/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 5311 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140829/3fbd300b/attachment.png>