<html><head></head><body bgcolor="#FFFFFF"><div>If you could provide the set up you were using that would be great. I can fill in anything missing an do my tests to verify.<br><br>On Mar 5, 2012, at 13:34, Emalayan Vairavanathan <<a href="mailto:svemalayan@yahoo.com">svemalayan@yahoo.com</a>> wrote:<br><br></div><div></div><blockquote type="cite"><div><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div><span>Thank you Jon. <br></span></div><div><br><span></span></div><div><span>Yesterday I successfully run Mosa (on our cluster) with cdm-direct mode with the help of swift-user manual and the scripts available in <span style="font-style: italic; font-weight: bold;"> /cog/modules/swift/tests/cdm/absolute</span>.</span></div><div><br><span></span></div><div><span>It would be useful if you can develop a simple test case. I can double check with my test case.</span></div><div><br><span></span></div><div><span>Thank you</span></div><div><span>Emalayan<br></span></div><div><br></div> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <font size="2" face="Arial"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Jonathan Monette <<a href="mailto:jonmon@mcs.anl.gov">jonmon@mcs.anl.gov</a>><br> <b><span style="font-weight: bold;">To:</span></b> Michael Wilde <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>> <br><b><span style="font-weight: bold;">Cc:</span></b> "<a href="mailto:emalayan@ece.ubc.ca">emalayan@ece.ubc.ca</a>" <<a href="mailto:emalayan@ece.ubc.ca">emalayan@ece.ubc.ca</a>>; "<a href="mailto:matei@ece.ubc.ca">matei@ece.ubc.ca</a>" <<a href="mailto:matei@ece.ubc.ca">matei@ece.ubc.ca</a>>; "<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>>; "<a href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>" <<a href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>>; Jonathan Monette <<a href="mailto:jon.monette@gmail.com">jon.monette@gmail.com</a>> <br> <b><span style="font-weight: bold;">Sent:</span></b> Monday, 5 March 2012 7:14 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Swift-devel] Configuring Swift to access MosaStore<br> </font> </div> <br>Yea. I will get demo scripts together for the mosa tests. <br><br>On Mar 5, 2012, at 8:17, Michael Wilde <<a ymailto="mailto:wilde@mcs.anl.gov" href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>> wrote:<br><br>> was:
Re: [Swift-devel] coasters-hosts.pl script<br>> <br>> Jon, can you create a demo script that shows how to configure a Swift run to use MosaStore. The following approach may work:<br>> <br>> - Assume MosaStore will be mounted as /mosa to all workers<br>> <br>> - Simulate this with a localhost run, using /tmp/mosa, then do same with *1* worker, N jobs per node (eg 4 on BG/P, 8 on PADS, 2 on Beagle).<br>> <br>> - Set CDM direct mode for all paths starting with [/tmp]/mosa. You might need to work through some of the issues with CDM direct where accesses need to match both /tmp/mosa and file:///tmp/mosa (I *think*)<br>> <br>> - Map some temporary output-to-input files to /tmp/mosa; create a multi-level "catsncats"-like workflow to exercise it; the recent ParameterSweep example, perhaps extended to do N levels of fan-in/fan-out and pass-N might be a good test.<br>> <br>> - see if you can get _concurrent to get placed on
/tmp/mosa<br>> <br>> I think some of these tests would be a great test case for Swift/Turbine as well.<br>> <br>> You can do this is stages; the simple test of mapping CDM-direct files to /tmp/mosa should give Emalayan an initial test case to run once Mosa is ready on the BG/P.<br>> <br>> - Mike<br>> <br>> <br>> ----- Original Message -----<br>>> From: "Matei Ripeanu" <<a ymailto="mailto:matei.ripeanu@gmail.com" href="mailto:matei.ripeanu@gmail.com">matei.ripeanu@gmail.com</a>><br>>> To: <a ymailto="mailto:mosastore@googlegroups.com" href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>, "Jonathan Monette" <<a ymailto="mailto:jonmon@mcs.anl.gov" href="mailto:jonmon@mcs.anl.gov">jonmon@mcs.anl.gov</a>>, "Justin M Wozniak" <<a ymailto="mailto:wozniak@mcs.anl.gov" href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>><br>>> Cc: <a ymailto="mailto:swift-devel@ci.uchicago.edu" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>, <a ymailto="mailto:emalayan@ece.ubc.ca" href="mailto:emalayan@ece.ubc.ca">emalayan@ece.ubc.ca</a><br>>> Sent: Friday, March 2, 2012 6:29:17 PM<br>>> Subject: Re: [Swift-devel] coasters-hosts.pl script<br>>> Indeed this is good news! Thank you.<br>>> <br>>> <br>>> <br>>> Our next task, I think, will be to figure out how to configure Swift<br>>> so that the headnode (where Swift runs) will not require any access to<br>>> intermediate storage (MosaStore). Only the worker nodes will have<br>>> access to intermediate storage. This is to go around the one way<br>>> headnode-worker node connectivity issue.<br>>> <br>>> <br>>> <br>>> Any guidance on how to get this configuration would be much<br>>> appreciated.<br>>> <br>>> <br>>>
<br>>> Thank you again,<br>>> <br>>> <br>>> <br>>> -Matei<br>>> <br>>> <br>>> <br>>> <br>>> <br>>> From: <a ymailto="mailto:mosastore@googlegroups.com" href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a> [mailto:<a ymailto="mailto:mosastore@googlegroups.com" href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>]<br>>> On Behalf Of Emalayan Vairavanathan<br>>> Sent: March-02-12 2:32 PM<br>>> To: Jonathan Monette; Justin M Wozniak<br>>> Cc: <a ymailto="mailto:swift-devel@ci.uchicago.edu" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> Devel; <a ymailto="mailto:emalayan@ece.ubc.cais" href="mailto:emalayan@ece.ubc.cais">emalayan@ece.ubc.cais</a> ;<br>>> MosaStore<br>>> Subject: Re: [Swift-devel] coasters-hosts.pl script<br>>> <br>>> <br>>> <br>>> <br>>>
<br>>> Thank you Jon and Justin.<br>>> <br>>> <br>>> <br>>> <br>>> <br>>> This is a great news. I will get back to you if I have questions.<br>>> <br>>> <br>>> <br>>> <br>>> <br>>> Regards<br>>> <br>>> <br>>> Emalayan<br>>> <br>>> <br>>> <br>>> <br>>> <br>>> <br>>> <br>>> <br>>> <br>>> From: Jonathan Monette < <a ymailto="mailto:jonmon@mcs.anl.gov" href="mailto:jonmon@mcs.anl.gov">jonmon@mcs.anl.gov</a> ><br>>> To: Justin M Wozniak < <a ymailto="mailto:wozniak@mcs.anl.gov" href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a> ><br>>> Cc: " <a ymailto="mailto:swift-devel@ci.uchicago.edu" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> Devel " <<br>>> <a ymailto="mailto:swift-devel@ci.uchicago.edu" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> >; <a ymailto="mailto:emalayan@ece.ubc.ca" href="mailto:emalayan@ece.ubc.ca">emalayan@ece.ubc.ca</a><br>>> Sent: Friday, 2 March 2012 2:21 PM<br>>> Subject: Re: [Swift-devel] coasters-hosts.pl script<br>>> <br>>> <br>>> Emalayan,<br>>> We believe we have fixed the issue. You can copy the new<br>>> coasters-hosts.pl script from<br>>> ~jonmon/surveyor/worker-init-test/coasters-hosts.pl<br>>> <br>>> This script reads the worker logs located in the logs directory. The<br>>> steps to run are as follows:<br>>> start-coaster-service<br>>> <wait for workers to start><br>>> ./coasters-hosts.pl logs/worker-*.log > worker-hosts.txt<br>>> <br>>> You MUST clean out the worker logs after you before you start a new<br>>> coaster service to make sure the script searches the right
worker log<br>>> files. This may not be ideal at the moment but this will help get you<br>>> started. If you have any other questions feel free to ask. We will<br>>> need to update the mosaswift site with the new information, we will do<br>>> this soon.<br>>> <br>>> On Mar 2, 2012, at 11:26 AM, Jonathan Monette wrote:<br>>> <br>>>> Can we match this line: 2012/03/02 17:16:04.712 INFO - Running on<br>>>> node 172.18.1.83 from the worker log,<br>>>> instead of this line: 2012-03-02 17:21:25,214+0000 DEBUG Cpu worker<br>>>> started: block=2012.0302.171344.704 host=172.18.1.83 id=0 from the<br>>>> cps log?<br>>>> <br>>>> They both provide the same ip addresses. And the worker log always<br>>>> has that ip address before the cps log does.<br>>>> <br>>>> On Mar 2, 2012, at 11:15 AM, Jonathan Monette wrote:<br>>>>
<br>>>>> That fix still did not work. I had moved it to the same spot. It is<br>>>>> still waiting for the worker-init.pl script to finish before the ip<br>>>>> addresses are printed to the cps log. Those ip addresses are what<br>>>>> is needed by the coaster-hosts.pl script to finish. If I create an<br>>>>> empty file for the coaster-host.pl script to read, then the work<br>>>>> continues and the ip addresses show up in the cps log.<br>>>>> <br>>>>> Why is log4j waiting to add those lines to the cps log after the<br>>>>> worker-init.pl script is finished?<br>>>>> <br>>>>> On Mar 2, 2012, at 11:05 AM, Jonathan Monette wrote:<br>>>>> <br>>>>>> Thanks, in my copy I thought I had moved the reconnect to before<br>>>>>> the init-cmd and it still wasn't working. I will test with
your<br>>>>>> change. I just verified that it was indeed waiting for the<br>>>>>> worker-init.pl script to finish. I created an empty file for the<br>>>>>> script to read and it finished connecting and the ip addresses I<br>>>>>> needed were added to the cps log. I will also be testing your fix.<br>>>>>> <br>>>>>> On Mar 2, 2012, at 11:01 AM, Justin M Wozniak wrote:<br>>>>>> <br>>>>>>> <br>>>>>>> Yes- I must have tested this with a different log file. I just<br>>>>>>> checked in and installed in ~wozniak/Public a fix for this that<br>>>>>>> launches WORKER_INIT_CMD after the reconnect(). I am a little<br>>>>>>> worried about time outs but it works so far. I will continue<br>>>>>>> testing...<br>>>>>>>
Justin<br>>>>>>> <br>>>>>>> On Thu, 1 Mar 2012, Jonathan Monette wrote:<br>>>>>>> <br>>>>>>>> Justin,<br>>>>>>>> So I have been trying to help Emalayan get the host list file<br>>>>>>>> for the worker-init.pl script. It seems the cps log file is not<br>>>>>>>> providing the ip addresses for the coasters-hosts.pl script. I<br>>>>>>>> thought this was maybe because we did not have the correct log4j<br>>>>>>>> setting set but we have the Coaster service Cpu set to DEBUG. So<br>>>>>>>> for some reason the workers are not connecting to the service.<br>>>>>>>> When I comment out the export WORKER_ENVIRONEMTN="…" line in the<br>>>>>>>> coaster-service.conf file I see the workers connect and the
cps<br>>>>>>>> log file shows there ip addresses. However when setting this<br>>>>>>>> line it seems they are not connecting.<br>>>>>>>> <br>>>>>>>> Emalayan thought there might be some sort of circular dependency<br>>>>>>>> going with the host-list file and the worker. The worker<br>>>>>>>> requires the host-list file so that it can run the<br>>>>>>>> worker-init.pl script and then connect but the host-list file<br>>>>>>>> cannot be generated because the workers cannot connect. I<br>>>>>>>> noticed in your swift-test directory the cps files did have the<br>>>>>>>> ip addresses set and coasters-hosts.pl found the ip addresses<br>>>>>>>> and reported them. Did you try that test with setting the<br>>>>>>>>
WORKER_ENVIRONMENT variable in the coaster-service.conf file?<br>>>>>>>> Any idea what may be happening? The job is running when looking<br>>>>>>>> under cqstat.<br>>>>>>>> <br>>>>>>>> A side note: At the mosaswift site, your example talks about<br>>>>>>>> running the coasters-hosts.pl on the cps log but the example you<br>>>>>>>> provide runs it on logs/coasters.log. This may need to be<br>>>>>>>> changed. Also, should provide the log4j setting that is required<br>>>>>>>> to generate the Cpu line with the worker ip address just to<br>>>>>>>> clarify that this line should be set for this script to work.<br>>>>>>>> <br>>>>>>>> For reference, this line:<br>>>>>>>>
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG<br>>>>>>> <br>>>>>>> --<br>>>>>>> Justin M Wozniak<br>>>>>> <br>>>>>> _______________________________________________<br>>>>>> Swift-devel mailing list<br>>>>>> <a ymailto="mailto:Swift-devel@ci.uchicago.edu" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>>>>>> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>>>>> <br>>>>> _______________________________________________<br>>>>> Swift-devel mailing list<br>>>>> <a ymailto="mailto:Swift-devel@ci.uchicago.edu" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>>>>> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>>>> <br>>>> _______________________________________________<br>>>> Swift-devel mailing list<br>>>> <a ymailto="mailto:Swift-devel@ci.uchicago.edu" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>>>> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>>> <br>>> <br>>> <br>>> <br>>> --<br>>> You received this message because you are subscribed to the Google<br>>> Groups "MosaStore" group.<br>>> To post to this group, send email to <a ymailto="mailto:mosastore@googlegroups.com" href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a> .<br>>>
To unsubscribe from this group, send email to<br>>> mosastore+<a ymailto="mailto:unsubscribe@googlegroups.com" href="mailto:unsubscribe@googlegroups.com">unsubscribe@googlegroups.com</a> .<br>>> For more options, visit this group at<br>>> <a href="http://groups.google.com/group/mosastore?hl=en" target="_blank">http://groups.google.com/group/mosastore?hl=en</a> .<br>>> _______________________________________________<br>>> Swift-devel mailing list<br>>> <a ymailto="mailto:Swift-devel@ci.uchicago.edu" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>>> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>> <br>> -- <br>> Michael Wilde<br>> Computation Institute, University of Chicago<br>> Mathematics and Computer Science Division<br>> Argonne National
Laboratory<br>> <br>_______________________________________________<br>Swift-devel mailing list<br><a ymailto="mailto:Swift-devel@ci.uchicago.edu" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br><a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br><br><br> </div> </div> </div></div></blockquote></body></html>