<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div><span>Forwarding Matei's mail.......</span></div><div><br><span></span></div><div><br></div> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <font size="2" face="Arial"> ----- Forwarded Message -----<br> <b><span style="font-weight:bold;">From:</span></b> Matei Ripeanu <matei.ripeanu@gmail.com><br> <b><span style="font-weight: bold;">To:</span></b> mosastore@googlegroups.com; 'Jonathan Monette' <jonmon@mcs.anl.gov>; 'Justin M Wozniak' <wozniak@mcs.anl.gov> <br><b><span style="font-weight: bold;">Cc:</span></b> swift-devel@ci.uchicago.edu; emalayan@ece.ubc.ca <br> <b><span style="font-weight: bold;">Sent:</span></b> Friday, 2 March 2012 4:29 PM<br> <b><span
style="font-weight: bold;">Subject:</span></b> RE: [Swift-devel] coasters-hosts.pl script<br> </font> </div> <br><div id="yiv942557829"><style><!--
#yiv942557829
_filtered #yiv942557829 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}
_filtered #yiv942557829 {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}
#yiv942557829
#yiv942557829 p.yiv942557829MsoNormal, #yiv942557829 li.yiv942557829MsoNormal, #yiv942557829 div.yiv942557829MsoNormal
{margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;font-family:"serif";}
#yiv942557829 a:link, #yiv942557829 span.yiv942557829MsoHyperlink
{color:blue;text-decoration:underline;}
#yiv942557829 a:visited, #yiv942557829 span.yiv942557829MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}
#yiv942557829 p
{margin-right:0cm;margin-left:0cm;font-size:12.0pt;font-family:"serif";}
#yiv942557829 p.yiv942557829MsoAcetate, #yiv942557829 li.yiv942557829MsoAcetate, #yiv942557829 div.yiv942557829MsoAcetate
{margin:0cm;margin-bottom:.0001pt;font-size:8.0pt;font-family:"sans-serif";}
#yiv942557829 span.yiv942557829EmailStyle18
{font-family:"sans-serif";color:#1F497D;}
#yiv942557829 span.yiv942557829BalloonTextChar
{font-family:"sans-serif";}
#yiv942557829 .yiv942557829MsoChpDefault
{font-size:10.0pt;}
_filtered #yiv942557829 {margin:72.0pt 72.0pt 72.0pt 72.0pt;}
#yiv942557829 div.yiv942557829WordSection1
{}
--></style><div><div class="yiv942557829WordSection1"><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;">Indeed this is good news! Thank you.</span></div><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;"> </span></div><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;">Our next task, I think, will be to figure out how to configure Swift so that the headnode (where Swift runs) will not require any access to intermediate storage (MosaStore). Only the worker nodes will have access to intermediate storage. This is to go around the one way headnode-worker node connectivity issue. </span></div><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;"> </span></div><div
class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;">Any guidance on how to get this configuration would be much appreciated.</span></div><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;"> </span></div><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;">Thank you again, </span></div><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;"> </span></div><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;">-Matei</span></div><div class="yiv942557829MsoNormal"><span style="font-size:11.0pt;font-family:"sans-serif";color:#1F497D;"> </span></div><div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm
0cm;"><div class="yiv942557829MsoNormal"><b><span style="font-size:10.0pt;font-family:"sans-serif";" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"sans-serif";" lang="EN-US"> mosastore@googlegroups.com [mailto:mosastore@googlegroups.com] <b>On Behalf Of </b>Emalayan Vairavanathan<br><b>Sent:</b> March-02-12 2:32 PM<br><b>To:</b> Jonathan Monette; Justin M Wozniak<br><b>Cc:</b> swift-devel@ci.uchicago.edu Devel; emalayan@ece.ubc.cais ; MosaStore<br><b>Subject:</b> Re: [Swift-devel] coasters-hosts.pl script</span></div></div></div><div class="yiv942557829MsoNormal"> </div><div><div><div class="yiv942557829MsoNormal" style="background:white;"><span style="color:black;">Thank you Jon and Justin. </span></div></div><div><div class="yiv942557829MsoNormal" style="background:white;"><span style="color:black;"> </span></div></div><div><div class="yiv942557829MsoNormal" style="background:white;"><span
style="color:black;">This is a great news. I will get back to you if I have questions.</span></div></div><div><div class="yiv942557829MsoNormal" style="background:white;"><span style="color:black;"> </span></div></div><div><div class="yiv942557829MsoNormal" style="background:white;"><span style="color:black;">Regards</span></div></div><div><div class="yiv942557829MsoNormal" style="background:white;"><span style="color:black;">Emalayan</span></div></div><div><div class="yiv942557829MsoNormal" style="background:white;"><span style="color:black;"> </span></div></div><div><div><div><div class="yiv942557829MsoNormal" style="text-align:center;background:white;" align="center"><span style="font-size:10.0pt;font-family:"sans-serif";color:black;"><hr size="1" width="100%" align="center"></span></div><div class="yiv942557829MsoNormal" style="background:white;"><b><span
style="font-size:10.0pt;font-family:"sans-serif";color:black;">From:</span></b><span style="font-size:10.0pt;font-family:"sans-serif";color:black;"> Jonathan Monette <<a rel="nofollow" ymailto="mailto:jonmon@mcs.anl.gov" target="_blank" href="mailto:jonmon@mcs.anl.gov">jonmon@mcs.anl.gov</a>><br><b>To:</b> Justin M Wozniak <<a rel="nofollow" ymailto="mailto:wozniak@mcs.anl.gov" target="_blank" href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>> <br><b>Cc:</b> "<a rel="nofollow" ymailto="mailto:swift-devel@ci.uchicago.edu%20Devel" target="_blank" href="mailto:swift-devel@ci.uchicago.edu%20Devel">swift-devel@ci.uchicago.edu Devel</a>" <<a rel="nofollow" ymailto="mailto:swift-devel@ci.uchicago.edu" target="_blank" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>>; <a rel="nofollow" ymailto="mailto:emalayan@ece.ubc.ca" target="_blank"
href="mailto:emalayan@ece.ubc.ca">emalayan@ece.ubc.ca</a> <br><b>Sent:</b> Friday, 2 March 2012 2:21 PM<br><b>Subject:</b> Re: [Swift-devel] coasters-hosts.pl script</span><span style="color:black;"></span></div></div><div class="yiv942557829MsoNormal" style="margin-bottom:12.0pt;background:white;"><span style="color:black;"><br>Emalayan,<br> We believe we have fixed the issue. You can copy the new coasters-hosts.pl script from ~jonmon/surveyor/worker-init-test/coasters-hosts.pl<br><br>This script reads the worker logs located in the logs directory. The steps to run are as follows:<br>start-coaster-service<br><wait for workers to start><br>./coasters-hosts.pl logs/worker-*.log > worker-hosts.txt<br><br>You MUST clean out the worker logs after you before you start a new coaster service to make sure the script searches the right worker log files. This may not be ideal at the moment but this will help get you
started. If you have any other questions feel free to ask. We will need to update the mosaswift site with the new information, we will do this soon.<br><br>On Mar 2, 2012, at 11:26 AM, Jonathan Monette wrote:<br><br>> Can we match this line: 2012/03/02 17:16:04.712 INFO - Running on node 172.18.1.83 from the worker log,<br>> instead of this line: 2012-03-02 17:21:25,214+0000 DEBUG Cpu worker started: block=2012.0302.171344.704 host=172.18.1.83 id=0 from the cps log?<br>> <br>> They both provide the same ip addresses. And the worker log always has that ip address before the cps log does.<br>> <br>> On Mar 2, 2012, at 11:15 AM, Jonathan Monette wrote:<br>> <br>>> That fix still did not work. I had moved it to the same spot. It is still waiting for the worker-init.pl script to finish before the ip addresses are printed to the cps log. Those ip addresses are what is needed by the
coaster-hosts.pl script to finish. If I create an empty file for the coaster-host.pl script to read, then the work continues and the ip addresses show up in the cps log. <br>>> <br>>> Why is log4j waiting to add those lines to the cps log after the worker-init.pl script is finished?<br>>> <br>>> On Mar 2, 2012, at 11:05 AM, Jonathan Monette wrote:<br>>> <br>>>> Thanks, in my copy I thought I had moved the reconnect to before the init-cmd and it still wasn't working. I will test with your change. I just verified that it was indeed waiting for the worker-init.pl script to finish. I created an empty file for the script to read and it finished connecting and the ip addresses I needed were added to the cps log. I will also be testing your fix.<br>>>> <br>>>> On Mar 2, 2012, at 11:01 AM, Justin M Wozniak wrote:<br>>>> <br>>>>> <br>>>>>
Yes- I must have tested this with a different log file. I just checked in and installed in ~wozniak/Public a fix for this that launches WORKER_INIT_CMD after the reconnect(). I am a little worried about time outs but it works so far. I will continue testing...<br>>>>> Justin<br>>>>> <br>>>>> On Thu, 1 Mar 2012, Jonathan Monette wrote:<br>>>>> <br>>>>>> Justin,<br>>>>>> So I have been trying to help Emalayan get the host list file for the worker-init.pl script. It seems the cps log file is not providing the ip addresses for the coasters-hosts.pl script. I thought this was maybe because we did not have the correct log4j setting set but we have the Coaster service Cpu set to DEBUG. So for some reason the workers are not connecting to the service. When I comment out the export WORKER_ENVIRONEMTN="…" line in the
coaster-service.conf file I see the workers connect and the cps log file shows there ip addresses. However when setting this line it seems they are not connecting.<br>>>>>> <br>>>>>> Emalayan thought there might be some sort of circular dependency going with the host-list file and the worker. The worker requires the host-list file so that it can run the worker-init.pl script and then connect but the host-list file cannot be generated because the workers cannot connect. I noticed in your swift-test directory the cps files did have the ip addresses set and coasters-hosts.pl found the ip addresses and reported them. Did you try that test with setting the WORKER_ENVIRONMENT variable in the coaster-service.conf file? Any idea what may be happening? The job is running when looking under cqstat.<br>>>>>> <br>>>>>> A side note: At the mosaswift site, your example
talks about running the coasters-hosts.pl on the cps log but the example you provide runs it on logs/coasters.log. This may need to be changed. Also, should provide the log4j setting that is required to generate the Cpu line with the worker ip address just to clarify that this line should be set for this script to work.<br>>>>>> <br>>>>>> For reference, this line: log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG<br>>>>> <br>>>>> -- <br>>>>> Justin M Wozniak<br>>>> <br>>>> _______________________________________________<br>>>> Swift-devel mailing list<br>>>> <a rel="nofollow" ymailto="mailto:Swift-devel@ci.uchicago.edu" target="_blank" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>>>> <a rel="nofollow" target="_blank"
href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>>> <br>>> _______________________________________________<br>>> Swift-devel mailing list<br>>> <a rel="nofollow" ymailto="mailto:Swift-devel@ci.uchicago.edu" target="_blank" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>>> <a rel="nofollow" target="_blank" href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>> <br>> _______________________________________________<br>> Swift-devel mailing list<br>> <a rel="nofollow" ymailto="mailto:Swift-devel@ci.uchicago.edu" target="_blank" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>> <a rel="nofollow" target="_blank"
href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br><br><br></span></div></div></div></div><div class="yiv942557829MsoNormal">-- <br>You received this message because you are subscribed to the Google Groups "MosaStore" group.<br>To post to this group, send email to <a rel="nofollow" ymailto="mailto:mosastore@googlegroups.com" target="_blank" href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>.<br>To unsubscribe from this group, send email to <a rel="nofollow" ymailto="mailto:mosastore+unsubscribe@googlegroups.com" target="_blank" href="mailto:mosastore+unsubscribe@googlegroups.com">mosastore+unsubscribe@googlegroups.com</a>.<br>For more options, visit this group at <a rel="nofollow" target="_blank" href="http://groups.google.com/group/mosastore?hl=en">http://groups.google.com/group/mosastore?hl=en</a>.</div></div></div></div><br><br>
</div> </div> </div></body></html>