[Swift-devel] auto-coaster bootstrap for stampede cluster

[Michael Wilde]

I'm moving this topic to swift-devel, so others, in particular Mihael, can weigh in.

- Mike

----- Forwarded Message -----
From: "David Kelly" <davidk at ci.uchicago.edu>
To: "Ketan Maheshwari" <ketan at mcs.anl.gov>
Cc: "Wilde" <wilde at mcs.anl.gov>
Sent: Wednesday, April 17, 2013 3:45:30 PM
Subject: Fwd: auto-coaster bootstrap for stamped

Hey Ketan, 

Mike mentioned that you were interested in running remotely to Stampede via ssh-cl. Normally we could use ssh-cl like any other site, but the problem we ran into here is that we can't run Swift on the stampede head node. We need to ssh-cl AND also start swift on a remote worker node, which is a setup that hasn't been tested very much. 

I believe you've used start-coaster-service before when we were running on ec2. You can this configuration for Stampede too. Modify coaster-service.conf to set WORKER_NODE=slurm, WORKER_RELAY_HOST=stampede.tacc.utexas.edu, and it will generate a slurm script, scp it to stampede, and remotely start swift on a worker node. I'll see if I can find an example config file for this. 

With automatic coaters it's a bit more complicated and completely untested as far as I know. 

You may be able to use gram2. This worked on Ranger, but haven't tried yet on Stampede. 
Mike mentioned in the email below you may be able to change the ssh-cl provider to add some kind of prefix command (srun). 
Maybe you can modify your PATH so the 'ssh' command is actually a wrapper you created and does something sneaky. 
You may also be able to add a prefix command to cog/modules/provider-coaster/resources/bootstrap.sh. 

Hopefully this can help you get started - let me know if any of this works for you, curious to see how we can get it working well. 

David 

----- Forwarded Message -----


From: "Michael Wilde" <wilde at mcs.anl.gov> 
To: "David Kelly" <davidk at ci.uchicago.edu> 
Sent: Tuesday, April 16, 2013 10:59:22 AM 
Subject: auto-coaster bootstrap for stamped 


was: Re: Another item for the to-do list 

David, thanks for the details. 

Im wondering, for systems like stampede, could automatic coasters work to it (eg from swift.rcc) by adding a sinteractive or srun command into the middle of the ssh command generated by the ssh-cl parameter? 

ie instead of doing ssh -sshargsgere auto-boostrap-coaster-stuff-here.sh 
do: ssh -sshargsgere srun auto-boostrap-coaster-stuff-here.sh 

? 

> This is the only mode that I've been able to test on Stampede so far. 
> I will experiment more the others when Stampede is back up. 

Others meaning GRAM? Perhaps using myproxy-logon? That *should* work out of the box but we've not tested GRAM in ages so it probably doesnt. 

Lets keep this lower on the prio list. I just want to be sure we have a ticket for this. Please create one if not - thanks.