<div dir="ltr">Thanks David. This sounds very useful for Stampede. I will try it for the remaining VASP runs on Stampede.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 24, 2013 at 3:51 PM, David Kelly <span dir="ltr"><<a href="mailto:davidk@ci.uchicago.edu" target="_blank">davidk@ci.uchicago.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif">Ketan,<div><br></div><div>I have gram working to Stampede now. Given the restrictions about running swift on the head nodes, I think this is the way to go. <span style="font-size:12pt">I'll add this info to the site guide, but for now here is a quick overview of what's needed.</span></div>
<div><br></div><div>Get a proxy: myproxy-logon -l username -s <a href="http://myproxy.teragrid.org" target="_blank">myproxy.teragrid.org</a></div><div><br></div><div>Make sure you have GLOBUS_HOSTNAME and GLOBUS_TCP_PORT_RANGE defined correctly.</div>
<div><br></div><div>Use something like this for your sites<span style="font-size:12pt">.xml (with work directory, project, and throttle adjusted as needed)</span></div><div><span style="font-size:12pt">---</span></div><div>
<div><div><config></div><div> <pool handle="stampede"></div><div> <execution provider="coaster" jobmanager="gt2:gt2:slurm" url="<a href="http://login5.stampede.tacc.utexas.edu:2119/jobmanager-slurm" target="_blank">login5.stampede.tacc.utexas.edu:2119/jobmanager-slurm</a>"/></div>
<div> <filesystem provider="gsiftp" url="gsiftp://<a href="http://gridftp.stampede.tacc.utexas.edu:2811" target="_blank">gridftp.stampede.tacc.utexas.edu:2811</a>"/></div><div> <profile namespace="globus" key="jobsPerNode">16</profile></div>
<div> <profile namespace="globus" key="ppn">16</profile></div><div> <profile namespace="globus" key="maxTime">3600</profile></div><div> <profile namespace="globus" key="maxwalltime">00:05:00</profile></div>
<div> <profile namespace="globus" key="lowOverallocation">100</profile></div><div> <profile namespace="globus" key="highOverallocation">100</profile></div>
<div> <profile namespace="globus" key="queue">normal</profile></div><div> <profile namespace="globus" key="nodeGranularity">1</profile></div><div> <profile namespace="globus" key="maxNodes">1</profile></div>
<div> <profile namespace="globus" key="project">TG-EAR130015</profile></div><div> <profile namespace="karajan" key="jobThrottle">.3199</profile></div>
<div> <profile namespace="karajan" key="initialScore">10000</profile></div><div> <workdirectory>/scratch/01503/davidkel</workdirectory></div><div> </pool></div><div>
</config></div></div><div>---</div><div><br></div><div>You'll also need the latest version of Swift from SVN. Swift was setting some invalid gram RSL attributes that were causing jobs to fail. I added a check to verify only valid attributes get set now. I've tested this with a simple swift script that calls /bin/hostname and it ran across multiple Stampede nodes. I haven't tested it with any larger applications yet - please let me know if you run into any problems with it.</div>
<div><br></div><div>Thanks,</div><div>David</div><br><hr><blockquote style="padding-left:5px;font-size:12pt;font-style:normal;margin-left:5px;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal;border-left:2px solid #1010ff">
<b>From: </b>"Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>><br><b>To: </b>"Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu" target="_blank">swift-devel@ci.uchicago.edu</a>><br>
<b>Sent: </b>Wednesday, April 17, 2013 3:51:31 PM<br><b>Subject: </b>[Swift-devel] auto-coaster bootstrap for stampede cluster<div><div class="h5"><br><br><br>I'm moving this topic to swift-devel, so others, in particular Mihael, can weigh in.<br>
<br>- Mike<br><br>----- Forwarded Message -----<br>From: "David Kelly" <<a href="mailto:davidk@ci.uchicago.edu" target="_blank">davidk@ci.uchicago.edu</a>><br>To: "Ketan Maheshwari" <<a href="mailto:ketan@mcs.anl.gov" target="_blank">ketan@mcs.anl.gov</a>><br>
Cc: "Wilde" <<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>><br>Sent: Wednesday, April 17, 2013 3:45:30 PM<br>Subject: Fwd: auto-coaster bootstrap for stamped<br><br>Hey Ketan, <br>
<br>Mike mentioned that you were interested in running remotely to Stampede via ssh-cl. Normally we could use ssh-cl like any other site, but the problem we ran into here is that we can't run Swift on the stampede head node. We need to ssh-cl AND also start swift on a remote worker node, which is a setup that hasn't been tested very much. <br>
<br>I believe you've used start-coaster-service before when we were running on ec2. You can this configuration for Stampede too. Modify coaster-service.conf to set WORKER_NODE=slurm, WORKER_RELAY_HOST=<a href="http://stampede.tacc.utexas.edu" target="_blank">stampede.tacc.utexas.edu</a>, and it will generate a slurm script, scp it to stampede, and remotely start swift on a worker node. I'll see if I can find an example config file for this. <br>
<br>With automatic coaters it's a bit more complicated and completely untested as far as I know. <br><br>You may be able to use gram2. This worked on Ranger, but haven't tried yet on Stampede. <br>Mike mentioned in the email below you may be able to change the ssh-cl provider to add some kind of prefix command (srun). <br>
Maybe you can modify your PATH so the 'ssh' command is actually a wrapper you created and does something sneaky. <br>You may also be able to add a prefix command to cog/modules/provider-coaster/resources/bootstrap.sh. <br>
<br>Hopefully this can help you get started - let me know if any of this works for you, curious to see how we can get it working well. <br><br>David <br><br>----- Forwarded Message -----<br><br><br>From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>> <br>
To: "David Kelly" <<a href="mailto:davidk@ci.uchicago.edu" target="_blank">davidk@ci.uchicago.edu</a>> <br>Sent: Tuesday, April 16, 2013 10:59:22 AM <br>Subject: auto-coaster bootstrap for stamped <br><br>
<br>was: Re: Another item for the to-do list <br><br>David, thanks for the details. <br><br>Im wondering, for systems like stampede, could automatic coasters work to it (eg from swift.rcc) by adding a sinteractive or srun command into the middle of the ssh command generated by the ssh-cl parameter? <br>
<br>ie instead of doing ssh -sshargsgere auto-boostrap-coaster-stuff-here.sh <br>do: ssh -sshargsgere srun auto-boostrap-coaster-stuff-here.sh <br><br>? <br><br>> This is the only mode that I've been able to test on Stampede so far. <br>
> I will experiment more the others when Stampede is back up. <br><br>Others meaning GRAM? Perhaps using myproxy-logon? That *should* work out of the box but we've not tested GRAM in ages so it probably doesnt. <br>
<br>Lets keep this lower on the prio list. I just want to be sure we have a ticket for this. Please create one if not - thanks. <br><br>_______________________________________________<br>Swift-devel mailing list<br><a href="mailto:Swift-devel@ci.uchicago.edu" target="_blank">Swift-devel@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br></div></div></blockquote><br></div></div></div></blockquote>
</div><br><br clear="all"><div><br></div>-- <br><font face="'courier new', monospace">Ketan</font><br><br>
</div>