<div dir="ltr">I re-run my coaster to make sure I am sending you an updated log. Log is attached. The sites.xml is this now <div><div><br></div><div><config></div><div> <pool handle="persistent-coasters"></div>
<div> <execution provider="coaster-persistent"</div><div> url="<a href="http://localhost:53346" target="_blank">http://localhost:53346</a>"</div><div> jobmanager="local:local"/></div>
<div> <profile namespace="globus" key="workerManager">passive</profile></div><div> <profile namespace="globus" key="jobsPerNode">1</profile></div><div>
<profile key="jobThrottle" namespace="karajan">10</profile></div><div> <profile namespace="karajan" key="initialScore">10000</profile></div><div> <filesystem provider="local" url="none" /></div>
<div> <workdirectory>/scratchspace</workdirectory></div><div> </pool></div><div></config></div></div><div><br></div><div><br></div><div>The first question is why the start-coaster-service does not terminate? </div>
<div><br></div><div>Anyhow, if I use this sites.xml, then swift output is </div><div><br></div><div>lenovo@lenovo-laptop:~/swift-cloud-tutorial/part01$ swift p1.swift</div><div>Swift 0.94.1 swift-r7114 cog-r3803</div><div>
<br></div><div>RunID: 20140902-1406-he5yo1s3</div><div>Progress: time: Tue, 02 Sep 2014 14:06:50 -0400</div><div>Progress: time: Tue, 02 Sep 2014 14:07:20 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:07:50 -0400 Submitted:1</div>
<div>Progress: time: Tue, 02 Sep 2014 14:08:20 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:08:50 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:09:20 -0400 Submitted:1</div><div>
Progress: time: Tue, 02 Sep 2014 14:09:50 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:10:20 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:10:50 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:11:20 -0400 Submitted:1</div>
<div>Progress: time: Tue, 02 Sep 2014 14:11:50 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:12:20 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:12:50 -0400 Submitted:1</div><div>
Progress: time: Tue, 02 Sep 2014 14:13:20 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:13:50 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:14:20 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:14:50 -0400 Submitted:1</div>
<div>Progress: time: Tue, 02 Sep 2014 14:15:20 -0400 Submitted:1</div><div>Progress: time: Tue, 02 Sep 2014 14:15:50 -0400 Submitted:1</div><div><br></div><div><br></div><div>I am also attaching the log file for that. </div>
<div><br></div><div> </div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Sep 1, 2014 at 5:20 PM, Justin bbt <span dir="ltr"><<a href="mailto:justinbbt@gmail.com" target="_blank">justinbbt@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I solved the permission problem with ssh-add command to add the key to list of keys. (This modification is required if the local system is linux- i am using ubuntu)<div>
<br></div><div>(more here <a href="https://help.github.com/articles/error-agent-admitted-failure-to-sign" target="_blank">https://help.github.com/articles/error-agent-admitted-failure-to-sign</a>)<br>
<div><br><div>Now, start-coaster-service connect to the cluster without password, but it does not terminate. The is the the output </div></div><div><br></div><div><div class=""><div>Service address: localhost</div><div>Starting coaster-service</div>
</div><div>Service port: 35925</div><div>Local port: 40681</div><div>Generating sites.xml</div><div>Starting worker on W.X.Y.Z</div><div>WORKER_LOGGING_LEVEL=DEBUG: Command not found.<br></div></div><div><br></div><div><br>
</div>
<div><br></div><div>If I just use my sites.xml </div><div><br></div><div><div class=""><div style="font-family:arial,sans-serif;font-size:12.7272720336914px"><div><pool handle="persistent-coasters"></div>
<div>
<execution provider="coaster-persistent"</div></div></div><div style="font-family:arial,sans-serif;font-size:12.7272720336914px"> url="<a href="http://localhost:37584/" target="_blank">http://</a>urladdress"</div>
<div class="">
<div style="font-family:arial,sans-serif;font-size:12.7272720336914px"><div> jobmanager="local:local"/></div><div> <profile namespace="globus" key="workerManager">passive</profile></div>
<div> <profile namespace="globus" key="jobsPerNode">1</profile></div><div> <profile key="jobThrottle" namespace="karajan">10</profile></div><div> <profile namespace="karajan" key="initialScore">10000</profile></div>
<div> <filesystem provider="local" url="none" /></div></div><div style="font-family:arial,sans-serif;font-size:12.7272720336914px"> <workdirectory>.</workdirectory></div><div style="font-family:arial,sans-serif;font-size:12.7272720336914px">
</pool></div></div></div><div><br></div><div><br></div><div> it fails with the following error </div><div><br></div><div><br></div><div><div class=""><div>Execution failed:</div><div><span style="white-space:pre-wrap"> </span>Exception in simulate:</div>
<div> Arguments: []</div></div><div> Host: persistent-coasters</div><div> Directory: p1-20140901-1648-r8mdqbse/jobs/z/simulate-zcyxesvl</div><div><br></div><div>Caused by:</div><div><span style="white-space:pre-wrap"> </span>Could not submit job</div>
<div>Caused by:</div><div><span style="white-space:pre-wrap"> </span>Failed to create socket</div><div>Caused by:</div><div><span style="white-space:pre-wrap"> </span>Connection refused</div><div><span style="white-space:pre-wrap"> </span>simulation, p1.swift, line 9</div>
</div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Aug 30, 2014 at 1:28 AM, Justin bbt <span dir="ltr"><<a href="mailto:justinbbt@gmail.com" target="_blank">justinbbt@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>For cluster: </div><div><br></div>When I run the start-caoster-service, I receive the following, in which it asks for password and then says Permission is denied <div>
<br></div><div><div>Start-coaster-service...</div>
<div>Configuration: /home/lenovo/swift-cloud-tutorial/scs/coaster-service.conf</div><div>Service address: localhost</div><div>Starting coaster-service</div><div>Service port: 52809</div><div>Local port: 58460</div><div>Generating sites.xml</div>
<div>username@ipadress's password: </div><div>username@ipadress's password: </div><div>Starting worker on username@</div><div>lenovo@lenovo-laptop:~/swift-cloud-tutorial/scs$username@ipadress's password: </div>
<div>Permission denied, please try again.</div><div>username@ipadress's password: </div><div>Permission denied, please try again.</div><div>username@ipadress's password: </div><div>Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).</div>
<div><br></div></div><div>This happens though I have created my keys with ssh-keygen. (only changed that I made was to create rsa keys rather than dsa keys - my cluster did not accept dsa). I can connect with rsa keygen and my passphrase for regular ssh</div>
<div><br></div><div>The output of my sites.xml from this partial running of start-coaster-service is </div><div><br></div><div><div><div> <pool handle="persistent-coasters"></div><div> <execution provider="coaster-persistent"</div>
</div><div> url="<a href="http://localhost:37584" target="_blank">http://localhost:37584</a>"</div><div><div> jobmanager="local:local"/></div><div> <profile namespace="globus" key="workerManager">passive</profile></div>
<div> <profile namespace="globus" key="jobsPerNode">1</profile></div><div> <profile key="jobThrottle" namespace="karajan">10</profile></div><div> <profile namespace="karajan" key="initialScore">10000</profile></div>
<div> <filesystem provider="local" url="none" /></div></div><div> <workdirectory>.</workdirectory></div><div> </pool></div></div><div><br></div><div>Using this XML , I just get a sequence of job submission every 30 seconds, no finished jobs. </div>
<div><br></div><div><br></div><div>BTW, I have a public ip for my cluster and then each compute node has a local/private ip. </div><div>In </div><div><div> <span style="font-family:arial,helvetica,sans-serif;font-size:12.7272720336914px">export WORKER_HOSTS="<IP of machine 1> <IP of machine 2>"</span></div>
</div><div><span style="font-family:arial,helvetica,sans-serif;font-size:12.7272720336914px">currently I just set the public IP address which still I am not successful with this one node even. I was wondering how should I set the other IPs? Does it mean that I have to install swift on the cluster? </span></div>
<div><span style="font-family:arial,helvetica,sans-serif;font-size:12.7272720336914px"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:12.7272720336914px"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:12.7272720336914px">I will look at the new release of swift for AWS. </span></div>
<div><span style="font-family:arial,helvetica,sans-serif;font-size:12.7272720336914px"><br></span></div><div><br></div><div>Thanks, </div><span><font color="#888888"><div>J. </div><div><br></div><div><br></div>
<div><br></div></font></span></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Aug 29, 2014 at 11:43 AM, Yadu Nand <span dir="ltr"><<a href="mailto:yadudoc1729@gmail.com" target="_blank">yadudoc1729@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi Justin,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
Did you do the following steps: </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">export WORKER_LOCATION="/home/ubuntu"</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
export WORKER_HOSTS="<IP of machine 1> <IP of machine 2>"</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">export WORKER_USERNAME=ubuntu</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">and then run "source setup.sh" ? </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">When you source the setup.sh scripts you must've gotten a sites.xml and a start-coaster-service.log in your scs folder, could you send us those ?</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The setup script should start a persistent coaster service and connect to the nodes on amazon, start workers, and generate a sites.xml file</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
that would let your swift scripts run across the amazon nodes. You shouldn't have to make changes to the sites.xml.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
Alternatively, you could try using the beta release of swift, Swift 0.95 RC6 with the new cloud mechanism:<br></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><a href="https://github.com/swift-lang/swift-on-cloud/tree/master/aws" target="_blank">https://github.com/swift-lang/swift-on-cloud/tree/master/aws</a></font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">That will set you up with a headnode on AWS with a few worker nodes that you define, with everything setup to run swift.</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
Thanks,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Yadu</div><div class="gmail_extra"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"></div><br><br><div class="gmail_quote">
<div><div>
On Thu, Aug 28, 2014 at 6:57 PM, Justin bbt <span dir="ltr"><<a href="mailto:justinbbt@gmail.com" target="_blank">justinbbt@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div>
<div><div><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hi all, <div><br></div><div>
I could successfully run swift on my local system. </div><div>Next, I want to use the swift to run some jobs on a cluster.<br></div><div><br></div><div>I followed this tutorial. (I am using just a simple cluster- I even could not run the job on one remote node of the cluster) </div>
<div><a href="http://swift-lang.org/tutorials/cloud/tutorial.html" target="_blank">http://swift-lang.org/tutorials/cloud/tutorial.html</a> <br></div><div><br></div><div>But, I get this when I run swift p1.swift or other swift </div>
<div>
<br></div><div>Swift 0.94.1 swift-r7114 cog-r3803</div><div><br></div><div>RunID: 20140828-1758-ea4phzag</div><div>Progress: time: Thu, 28 Aug 2014 17:58:15 -0400</div><div>Progress: time: Thu, 28 Aug 2014 17:58:24 -0400 Submitted:1</div>
<div>Execution failed:</div><div><span style="white-space:pre-wrap"> </span>Exception in simulate:</div><div> Arguments: []</div><div> Host: remotehost2</div><div> Directory: p1-20140828-1758-ea4phzag/jobs/7/simulate-7k2fxlvl</div>
<div><br></div><div>Caused by:</div><div><span style="white-space:pre-wrap"> </span>Job failed with an exit code of 127</div><div><span style="white-space:pre-wrap"> </span>simulation, p1.swift, line 9</div><div><br>
</div><div><br></div><div>--- this is my site.xml file setting </div><div><div><br></div><div> <pool handle="remotehost2"></div><div> <execution provider="ssh" jobmanager="ssh:local" url="myclusteturl"/></div>
<div> <filesystem provider="ssh" url="myclusteturl"/></div><div> <profile namespace="karajan" key="jobThrottle">0</profile></div><div> <profile namespace="karajan" key="initialScore">10000</profile></div>
<div> <workdirectory>/path/to/remote/workdirectory</workdirectory></div><div> </pool></div></div><div><br></div><div>--- if I use this one </div><div><div><pool handle="persistent-coasters"></div>
<div> <execution provider="coaster-persistent"</div><div> url="myclusterurl"</div><div> jobmanager="local:local"/></div><div> <profile namespace="globus" key="workerManager">passive</profile></div>
<div> <profile namespace="globus" key="jobsPerNode">1</profile></div><div> <profile key="jobThrottle" namespace="karajan">10</profile></div><div> <profile namespace="karajan" key="initialScore">10000</profile></div>
<div> <filesystem provider="local" url="none" /></div><div> <workdirectory>.l</workdirectory></div><div> </pool></div></div><div>--- then it loops to my localhost and just repeat submitting the jobs </div>
<div><br></div><div>1. Is this a correct setting? </div><div>2. Should I use coaster? I could not understand the description in user guides and documentation about the concepts of coaster and the required setting. Is there any better tutorial which would describe the coaster ? </div>
<div>3. I plan to use the swift later on the cloud (Microsoft Azure). What are the setting required for that? for site.xml and if any other file </div><div><br></div><div><br></div><div>Thanks in Advance. </div><div> </div>
</div>
</blockquote></div><br></div></div>
</div></div><br></div></div>_______________________________________________<br>
Swift-user mailing list<br>
<a href="mailto:Swift-user@ci.uchicago.edu" target="_blank">Swift-user@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><span><font color="#888888"><br></font></span></blockquote>
</div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br>
<div dir="ltr">Yadu Nand B<div><br></div></div>
</font></span></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>