<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div style="-webkit-text-size-adjust: auto;"><span></span></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="-webkit-text-size-adjust: auto;">Thanks Mike and Yadu,</div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">Yes it would be great if there is a published config for this use case, and I am willing to help in whatever capacity I can. </div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">The docker containers can reach out to external IP addresses, so as you both mention the best way may be to run the worker.pl in each container and have it connect back to the coaster service. I start the coaster service on a known ip/port host using the command you specify. </div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">I think I conceptually understand how this could work and will try to test something. </div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">I have an ubuntu docker container configured to start ssh services at the command below. It sounds like, however, if I can run the worker.pl on each container and connect to the coaster service, ssh is not even required. </div><div style="-webkit-text-size-adjust: auto;"><br></div><div><span style="-webkit-text-size-adjust: auto; background-color: rgba(255, 255, 255, 0);">docker run -i -p 22 -t mattshax/precise_ep_node sh -c '/usr/sbin/sshd -D'</span></div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;"><br>On Aug 29, 2014, at 9:33 PM, "Yadu Nand" <<a href="mailto:yadudoc1729@gmail.com">yadudoc1729@gmail.com</a>> wrote:<br><br></div><blockquote type="cite" style="-webkit-text-size-adjust: auto;"><div><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi Matthew,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
You can use the coaster-service directly with the following arguments :</div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">coaster-service -p $SERVICEPORT -localport $WORKERPORT -nosec -passive &> /var/log/coaster-service.logs</font><br>
</div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">The workers running under docker should be connecting to the WORKERPORT and the swift client would use the SERVICEPORT.</font></div>
<div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Another thing to remember is to make sure that you use the the coaster-service and <a href="http://worker.pl">worker.pl</a> from the same swift release. </font></div>
<div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">The start-coaster-service command is a wrapper over coaster-service which in addition to starting the coaster service, also </font></div>
<div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">tries to ssh the <a href="http://worker.pl">worker.pl</a> to the WORKER_HOSTS. In the situation that you describe, I think it makes sense to have the workers</font></div>
<div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">just connect back to the coaster-service which is listening on a known IP / port.</font></div><div class="gmail_default" style=""><br></div><div class="gmail_default" style="">
Thanks,</div><div class="gmail_default" style="">Yadu</div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br>
</font></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Aug 29, 2014 at 8:34 PM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@anl.gov" target="_blank">wilde@anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Matthew,<br>
<br>
I would treat this case similar to the configuration you'd use for a
set of virtual machines.<br>
<br>
Start one coaster service for each pool of identical docker
containers that you want to run.<br>
<br>
For each pool of containers, run a Swift worker (<a href="http://worker.pl" target="_blank">worker.pl</a>) in the
container and have it connect back to the coaster service you
designate to manage that pool.<br>
<br>
I'm assuming that from a docker contained you can connect out to any
reachable IP address. <br>
<br>
Another approach is to treat the containers like a set of ad-hoc
compute nodes, and ssh into them with automatic coasters using the
<a>ssh:local</a> jobmanager setting.<br>
<br>
We'll try to test and publish a config for such cases. We'd welcome
your help with that.<br>
<br>
- Mike<div><div class="h5"><br>
<br>
<br>
<div>On 8/29/14, 7:48 PM, Matthew Shaxted
wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<div>
<p class="MsoNormal">Hi All,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’m trying to find a way to run Swift
workflows on multi-host docker containers, and wondering if
anybody has had success with this. <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">When I start a docker container and define
specific ports to open on the container, they mapped to random
ports on the host machine. So for example, I can start a
container with an ssh port open from a host “10.1.1.1”, and
can then access this container across hosts with “ssh
<a href="mailto:compute@10.1.1.1" target="_blank">compute@10.1.1.1</a> –p 49160”.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Now I’m hoping to link these docker
containers to Swift’s start-coaster-service. I think it would
be possible and relatively easy if I can say, start the
coasters on a series of IP addresses AND ports. So a host’s
file perhaps would look something like below, and coasters
would be started on the correct docker container:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><a href="http://10.1.1.1" target="_blank">10.1.1.1</a>: 49160<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><a href="http://10.1.1.2" target="_blank">10.1.1.2</a>: 34155<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><a href="http://10.1.1.2" target="_blank">10.1.1.2</a>: 34156
…<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Does this make sense? Is it possible to
start coasters by specifying an IP address and port number?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Any thoughts are greatly appreciated.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Matthew<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-bottom:14.0pt;line-height:13.0pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#ef2b2d">MATTHEW
SHAXTED<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:14.0pt;line-height:13.0pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:gray">SKIDMORE,
OWINGS & MERRILL LLP<br>
224 SOUTH MICHIGAN AVENUE<br>
CHICAGO, IL 60604<br>
T <a href="tel:%28312%29%20360-4368" value="+13123604368" target="_blank">(312) 360-4368</a><br>
<a href="mailto:MATTHEW.SHAXTED@SOM.COM" target="_blank"><span style="color:blue">MATTHEW.SHAXTED@SOM.COM</span></a><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:14.0pt;line-height:13.0pt"><span style="font-family:"Arial","sans-serif";color:gray"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-bottom:14.0pt"><a href="http://www.som.com/" target="_blank"><span style="font-family:"Arial","sans-serif";color:black;text-decoration:none"><ATT00002.png></span></a><a name="148248c4c32ac371__GoBack"></a><span style="font-family:"Arial","sans-serif";color:black"><u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:12.0pt"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">The
information contained in this communication may be
confidential, is intended only for the use of the
recipient(s) named above, and may be legally privileged. If
the reader of this message is not the intended recipient,
you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of
its contents, is strictly prohibited and may be unlawful. If
you have received this communication in error, please return
it to the sender immediately and delete the original
message and any copy of it from your computer system. If you
have any questions concerning this message, please contact
the sender.</span><span style="font-family:"Arial","sans-serif";color:gray"><u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana","sans-serif";color:black"><ATT00001.gif></span><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><pre>_______________________________________________
Swift-user mailing list
<a href="mailto:Swift-user@ci.uchicago.edu" target="_blank">Swift-user@ci.uchicago.edu</a>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a></pre><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<pre cols="72">--
Michael Wilde
Mathematics and Computer Science Computation Institute
Argonne National Laboratory The University of Chicago
</pre>
</font></span></div>
<br>_______________________________________________<br>
Swift-user mailing list<br>
<a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br>
<div dir="ltr">Yadu Nand B<div><br></div></div>
</div>
</div></blockquote><blockquote type="cite" style="-webkit-text-size-adjust: auto;"><div><span>_______________________________________________</span><br><span>Swift-user mailing list</span><br><span><a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a></span><br><span><a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a></span></div></blockquote></div></body></html>