<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000'>Ketan,<div><br></div><div>I have gram working to Stampede now. Given the restrictions about running swift on the head nodes, I think this is the way to go. <span style="font-size: 12pt;">I'll add this info to the site guide, but for now here is a quick overview of what's needed.</span></div><div><br></div><div>Get a proxy: myproxy-logon -l username -s myproxy.teragrid.org</div><div><br></div><div>Make sure you have GLOBUS_HOSTNAME and GLOBUS_TCP_PORT_RANGE defined correctly.</div><div><br></div><div>Use something like this for your sites<span style="font-size: 12pt;">.xml (with work directory, project, and throttle adjusted as needed)</span></div><div><span style="font-size: 12pt;">---</span></div><div><div><div><config></div><div> <pool handle="stampede"></div><div> <execution provider="coaster" jobmanager="gt2:gt2:slurm" url="login5.stampede.tacc.utexas.edu:2119/jobmanager-slurm"/></div><div> <filesystem provider="gsiftp" url="gsiftp://gridftp.stampede.tacc.utexas.edu:2811"/></div><div> <profile namespace="globus" key="jobsPerNode">16</profile></div><div> <profile namespace="globus" key="ppn">16</profile></div><div> <profile namespace="globus" key="maxTime">3600</profile></div><div> <profile namespace="globus" key="maxwalltime">00:05:00</profile></div><div> <profile namespace="globus" key="lowOverallocation">100</profile></div><div> <profile namespace="globus" key="highOverallocation">100</profile></div><div> <profile namespace="globus" key="queue">normal</profile></div><div> <profile namespace="globus" key="nodeGranularity">1</profile></div><div> <profile namespace="globus" key="maxNodes">1</profile></div><div> <profile namespace="globus" key="project">TG-EAR130015</profile></div><div> <profile namespace="karajan" key="jobThrottle">.3199</profile></div><div> <profile namespace="karajan" key="initialScore">10000</profile></div><div> <workdirectory>/scratch/01503/davidkel</workdirectory></div><div> </pool></div><div></config></div></div><div>---</div><div><br></div><div>You'll also need the latest version of Swift from SVN. Swift was setting some invalid gram RSL attributes that were causing jobs to fail. I added a check to verify only valid attributes get set now. I've tested this with a simple swift script that calls /bin/hostname and it ran across multiple Stampede nodes. I haven't tested it with any larger applications yet - please let me know if you run into any problems with it.</div><div><br></div><div>Thanks,</div><div>David</div><br><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Michael Wilde" <wilde@mcs.anl.gov><br><b>To: </b>"Swift Devel" <swift-devel@ci.uchicago.edu><br><b>Sent: </b>Wednesday, April 17, 2013 3:51:31 PM<br><b>Subject: </b>[Swift-devel] auto-coaster bootstrap for stampede cluster<br><br><br>I'm moving this topic to swift-devel, so others, in particular Mihael, can weigh in.<br><br>- Mike<br><br>----- Forwarded Message -----<br>From: "David Kelly" <davidk@ci.uchicago.edu><br>To: "Ketan Maheshwari" <ketan@mcs.anl.gov><br>Cc: "Wilde" <wilde@mcs.anl.gov><br>Sent: Wednesday, April 17, 2013 3:45:30 PM<br>Subject: Fwd: auto-coaster bootstrap for stamped<br><br>Hey Ketan, <br><br>Mike mentioned that you were interested in running remotely to Stampede via ssh-cl. Normally we could use ssh-cl like any other site, but the problem we ran into here is that we can't run Swift on the stampede head node. We need to ssh-cl AND also start swift on a remote worker node, which is a setup that hasn't been tested very much. <br><br>I believe you've used start-coaster-service before when we were running on ec2. You can this configuration for Stampede too. Modify coaster-service.conf to set WORKER_NODE=slurm, WORKER_RELAY_HOST=stampede.tacc.utexas.edu, and it will generate a slurm script, scp it to stampede, and remotely start swift on a worker node. I'll see if I can find an example config file for this. <br><br>With automatic coaters it's a bit more complicated and completely untested as far as I know. <br><br>You may be able to use gram2. This worked on Ranger, but haven't tried yet on Stampede. <br>Mike mentioned in the email below you may be able to change the ssh-cl provider to add some kind of prefix command (srun). <br>Maybe you can modify your PATH so the 'ssh' command is actually a wrapper you created and does something sneaky. <br>You may also be able to add a prefix command to cog/modules/provider-coaster/resources/bootstrap.sh. <br><br>Hopefully this can help you get started - let me know if any of this works for you, curious to see how we can get it working well. <br><br>David <br><br>----- Forwarded Message -----<br><br><br>From: "Michael Wilde" <wilde@mcs.anl.gov> <br>To: "David Kelly" <davidk@ci.uchicago.edu> <br>Sent: Tuesday, April 16, 2013 10:59:22 AM <br>Subject: auto-coaster bootstrap for stamped <br><br><br>was: Re: Another item for the to-do list <br><br>David, thanks for the details. <br><br>Im wondering, for systems like stampede, could automatic coasters work to it (eg from swift.rcc) by adding a sinteractive or srun command into the middle of the ssh command generated by the ssh-cl parameter? <br><br>ie instead of doing ssh -sshargsgere auto-boostrap-coaster-stuff-here.sh <br>do: ssh -sshargsgere srun auto-boostrap-coaster-stuff-here.sh <br><br>? <br><br>> This is the only mode that I've been able to test on Stampede so far. <br>> I will experiment more the others when Stampede is back up. <br><br>Others meaning GRAM? Perhaps using myproxy-logon? That *should* work out of the box but we've not tested GRAM in ages so it probably doesnt. <br><br>Lets keep this lower on the prio list. I just want to be sure we have a ticket for this. Please create one if not - thanks. <br><br>_______________________________________________<br>Swift-devel mailing list<br>Swift-devel@ci.uchicago.edu<br>https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel<br></blockquote><br></div></div></body></html>