<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Microsoft Sans Serif";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="2050" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi David,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I tried out the cluster and it does seem to make the start-coaster-service much faster – however, I am getting VM heap space issues when starting swift on the configuration I am using so it is making it difficult to use the cluster head node.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Also, I have many of my workflows setup to read from the glusterfs so I may just stick with the slow coaster start for now. The bigger issue is that even when these start, they do not remain persistent for some reason (could be my localhost coasters). Do I need to explicitly express my localhost coasters as persistent in the sites.xml file?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> I have been getting strange issues when I try to scale my runs (10-12 hrs each) on the glusterfs configuration – they seem to be failing midway through for some reason although they work with smaller runs, so I’m going to put sometime into understanding why this is now.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Matthew<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:red;text-transform:uppercase'>MATTHEW SHAXTED</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;text-transform:uppercase'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:black;text-transform:uppercase'>SKIDMORE, OWINGS & MERRILL LLP </span><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#1F497D;text-transform:uppercase'> </span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;text-transform:uppercase'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>224 South Michigan Ave.</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#736F6E;text-transform:uppercase'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>Chicago, IL 60604</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#736F6E;text-transform:uppercase'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>TEL: 312.360.4368</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;text-transform:uppercase'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>FAX: 312.360.4545</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;text-transform:uppercase'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><a href="mailto:matthew.shaxted@som.com"><span style='font-size:7.5pt;text-transform:uppercase'>matthew.shaxted@som.com</span></a><span style='text-transform:uppercase'><o:p></o:p></span></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><img border=0 width=78 height=40 id="Picture_x0020_1" src="cid:image001.png@01CF2D6E.DB47EC10" alt="cid:image9d6458.png@2965c709.c87949ac"><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><a href="http://www.som.com/"><span style='font-size:7.5pt'>WWW.SOM.COM</span></a></span><span style='font-size:5.5pt;font-family:"Calibri","sans-serif";color:#1F497D'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#1F497D'>The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender.</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><img border=0 width=392 height=17 id="Picture_x0020_2" src="cid:image002.png@01CF2D6E.DB47EC10" alt="cid:image93798a.gif@f078f826.ddd94773"><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> David Kelly [mailto:davidkelly@uchicago.edu] <br><b>Sent:</b> Monday, February 17, 2014 7:21 PM<br><b>To:</b> Matthew Shaxted<br><b>Cc:</b> Wilde, Michael J.; swift-user@ci.uchicago.edu<br><b>Subject:</b> Re: [Swift-user] Coaster Service Startup Time on OSDC<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Sounds good, Matthew. Let me know how that works for you. I filed a ticket with osdc support about the SSH issue, so hopefully they can offer some help there. <o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I added an entry to our site guide documentation about how to run on OSDC in cluster mode. It is at <a href="http://swiftlang.org/guides/release-0.94/siteguide/siteguide.html#_open_science_data_grid">http://swiftlang.org/guides/release-0.94/siteguide/siteguide.html#_open_science_data_grid</a>. The only potential issue is, it seems to only work with the standard Ubuntu images. <o:p></o:p></p></div></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p><div><p class=MsoNormal>On Mon, Feb 17, 2014 at 3:34 PM, Matthew Shaxted <<a href="mailto:Matthew.Shaxted@som.com" target="_blank">Matthew.Shaxted@som.com</a>> wrote:<o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi David,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Indeed I am running start-coaster-service from the head node.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The first recommendation is a good one, I will try this out and let you know how it works.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I am also very interested in the cluster launch/PBS scheduler approach, although I have never used it before. An example OSDC/PBS config would be really helpful.</span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Matthew</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:red;text-transform:uppercase'>MATTHEW SHAXTED</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";text-transform:uppercase'>SKIDMORE, OWINGS & MERRILL LLP <span style='color:#1F497D'> </span></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>224 South Michigan Ave.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>Chicago, IL 60604</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>TEL: <a href="tel:312.360.4368" target="_blank">312.360.4368</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>FAX: <a href="tel:312.360.4545" target="_blank">312.360.4545</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><a href="mailto:matthew.shaxted@som.com" target="_blank"><span style='font-size:7.5pt;text-transform:uppercase'>matthew.shaxted@som.com</span></a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><img border=0 width=78 height=40 id="_x0000_i1025" src="cid:image001.png@01CF2D6E.DB47EC10" alt="cid:image9d6458.png@2965c709.c87949ac"></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><a href="http://www.som.com/" target="_blank"><span style='font-size:7.5pt'>WWW.SOM.COM</span></a></span><span style='font-size:5.5pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#1F497D'>The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><img border=0 width=392 height=17 id="_x0000_i1026" src="cid:image002.png@01CF2D6E.DB47EC10" alt="cid:image93798a.gif@f078f826.ddd94773"></span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Wilde, Michael J. [mailto:<a href="mailto:wilde@anl.gov" target="_blank">wilde@anl.gov</a>] <br><b>Sent:</b> Monday, February 17, 2014 2:36 PM<br><b>To:</b> David Kelly; Matthew Shaxted<br><b>Cc:</b> <a href="mailto:swift-user@ci.uchicago.edu" target="_blank">swift-user@ci.uchicago.edu</a><br><b>Subject:</b> RE: [Swift-user] Coaster Service Startup Time on OSDC</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Good find, David. Did you file a ticket on the slowness with OSDC Support?<br><br>When you run ssh -vvv, does the timing of log output suggest where the problem is?<br><br>Can you run into some tool like typescript, or a "screen" log, that will timestamp the records, and send those to OSDC Support?</span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> </span><o:p></o:p></p><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>- Mike </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>--</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Michael Wilde</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Mathematics and Computer Science Computation Institute</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Argonne National Laboratory The University of Chicago</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> </span><o:p></o:p></p></div></div></div></div></div></div></div><div><div class=MsoNormal align=center style='text-align:center'><hr size=2 width="100%" align=center></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:swift-user-bounces@ci.uchicago.edu" target="_blank">swift-user-bounces@ci.uchicago.edu</a> [<a href="mailto:swift-user-bounces@ci.uchicago.edu" target="_blank">swift-user-bounces@ci.uchicago.edu</a>] on behalf of David Kelly [<a href="mailto:davidkelly@uchicago.edu" target="_blank">davidkelly@uchicago.edu</a>]<br><b>Sent:</b> Monday, February 17, 2014 2:14 PM<br><b>To:</b> Matthew Shaxted<br><b>Cc:</b> <a href="mailto:swift-user@ci.uchicago.edu" target="_blank">swift-user@ci.uchicago.edu</a><br><b>Subject:</b> Re: [Swift-user] Coaster Service Startup Time on OSDC</span><o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Matthew, <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I set up a test on OSDC with 10 nodes. I did notice something strange there. When I try to SSH from the Sullivan head node to one of my CentOS instances, it takes much longer than it should:<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>dkelly@kg14-compute-1:~$ time ssh <a href="mailto:root@172.16.1.236" target="_blank">root@172.16.1.236</a> ls<br>Warning: Permanently added '172.16.1.236' (RSA) to the list of known hosts.<br>anaconda-ks.cfg<br>install.log<br>install.log.syslog <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br>real 0m25.197s<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Are you running start-coaster-service from the Sullivan head node? For each node in your node list, there are three SSH commands (one to create a directory structure, one to scp the <a href="http://worker.pl/" target="_blank">worker.pl</a> script there, and one to launch <a href="http://worker.pl/" target="_blank">worker.pl</a>). This is done serially in the 0.94 branch. I can see how this would take very long. I will make some changes to speed up start-coaster-service, but in the meantime, here are a few suggestions:<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>1. The SSH slowness seems to only be from the Sullivan head node to the VMs. SSH connections from one VM to another VM is pretty quick. Are you able to run Swift and start-coaster-service on a VM?<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>2. Is a persistent coasters setup needed here? OSDC has an option to launch instances as a cluster, which makes available a PBS scheduler. You could set this up and avoid the need to start and manage workers yourself. <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Let me know what you think. I have some example OSDC/PBS configs if you decide to go that route.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>David<o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Thu, Feb 13, 2014 at 1:04 PM, Matthew Shaxted <<a href="mailto:Matthew.Shaxted@som.com" target="_blank">Matthew.Shaxted@som.com</a>> wrote:<o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Sure thing David,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I have a setup.sh script that exports WORKER_HOSTS IP addresses from a txt file (see attached) – all nodes are running centOS. </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The conf file I am using is also attached. I’m using swift-0.94.1.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Many thanks,</span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Matthew</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:red;text-transform:uppercase'>MATTHEW SHAXTED</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";text-transform:uppercase'>SKIDMORE, OWINGS & MERRILL LLP <span style='color:#1F497D'> </span></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>224 South Michigan Ave.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>Chicago, IL 60604</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>TEL: <a href="tel:312.360.4368" target="_blank">312.360.4368</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>FAX: <a href="tel:312.360.4545" target="_blank">312.360.4545</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><a href="mailto:matthew.shaxted@som.com" target="_blank"><span style='font-size:7.5pt;text-transform:uppercase'>matthew.shaxted@som.com</span></a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><img border=0 width=78 height=40 id="_x0000_i1028" src="cid:image001.png@01CF2D6E.DB47EC10" alt="cid:image9d6458.png@2965c709.c87949ac"></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><a href="http://www.som.com/" target="_blank"><span style='font-size:7.5pt'>WWW.SOM.COM</span></a></span><span style='font-size:5.5pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#1F497D'>The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><img border=0 width=392 height=17 id="_x0000_i1029" src="cid:image002.png@01CF2D6E.DB47EC10" alt="cid:image93798a.gif@f078f826.ddd94773"></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> David Kelly [mailto:<a href="mailto:davidkelly@uchicago.edu" target="_blank">davidkelly@uchicago.edu</a>] <br><b>Sent:</b> Thursday, February 13, 2014 12:53 PM<br><b>To:</b> Matthew Shaxted<br><b>Cc:</b> <a href="mailto:swift-user@ci.uchicago.edu" target="_blank">swift-user@ci.uchicago.edu</a><br><b>Subject:</b> Re: [Swift-user] Coaster Service Startup Time on OSDC</span><o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Matthew,<o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Could you please explain a little more about how you're starting the coaster-service and workers? Are you using the start-coaster-service script? If you are, could you please send the coaster-service.conf file you're using? Which version of Swift is this? <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>There may be some things we can do to speed up the process - just need to get a better understanding of how things are set up and where the delays are coming from. Thanks!<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Regards,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>David<o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Thu, Feb 13, 2014 at 12:23 PM, Matthew Shaxted <<a href="mailto:Matthew.Shaxted@som.com" target="_blank">Matthew.Shaxted@som.com</a>> wrote:<o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Dear Swift User Group:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I working with a 60-node cluster running on OSDC now, and when I try to start the Swift coaster-service for these nodes, it takes about 30 seconds (or more) per node to successfully start. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>This is an issue for me as it limits how often I want to shut down the coaster-service - for this 60 node cluster it could take up to 30 min to start up again.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Is this starting coaster behavior normal? Is there anything I can do to make the coaster-service start faster?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks,<br>Matthew<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:red;text-transform:uppercase'>MATTHEW SHAXTED</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";text-transform:uppercase'>SKIDMORE, OWINGS & MERRILL LLP <span style='color:#1F497D'> </span></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>224 South Michigan Ave.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>Chicago, IL 60604</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>TEL: <a href="tel:312.360.4368" target="_blank">312.360.4368</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#736F6E;text-transform:uppercase'>FAX: <a href="tel:312.360.4545" target="_blank">312.360.4545</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'><a href="mailto:matthew.shaxted@som.com" target="_blank"><span style='font-size:7.5pt;text-transform:uppercase'>matthew.shaxted@som.com</span></a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'><img border=0 width=78 height=40 id="_x0000_i1030" src="cid:image001.png@01CF2D6E.DB47EC10" alt="cid:image9d6458.png@2965c709.c87949ac"></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'><a href="http://www.som.com/" target="_blank"><span style='font-size:7.5pt'>WWW.SOM.COM</span></a></span><span style='font-size:5.5pt;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Microsoft Sans Serif","sans-serif";color:#1F497D'>The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#1F497D'><img border=0 width=392 height=17 id="_x0000_i1031" src="cid:image002.png@01CF2D6E.DB47EC10" alt="cid:image93798a.gif@f078f826.ddd94773"></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br>_______________________________________________<br>Swift-user mailing list<br><a href="mailto:Swift-user@ci.uchicago.edu" target="_blank">Swift-user@ci.uchicago.edu</a><br><a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div></div></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div></div></div></div></div></div></div></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>