[Swift-devel] Re: Getting VMs from FG for use with swift

Tue Jun 7 18:59:15 CDT 2011

great!  I will take a look at that when I can (I am on travel right now).

Getting back to using cloudinit.d for swift... we you able to use the VMs you got from FutureGrid with swift?  I was hoping we could start encorperating that into swift, or at least acquire a set of VMs and use those as you currently use static machines in some demos.

----- Original Message -----
From: "David Kelly" <davidkelly999 at gmail.com>
To: "John Bresnahan" <bresnaha at mcs.anl.gov>
Cc: "Mike Wilde" <wilde at mcs.anl.gov>, "swift-devel" <swift-devel at ci.uchicago.edu>
Sent: Tuesday, June 7, 2011 2:39:20 PM
Subject: Re: Getting VMs from FG for use with swift

Hello John, 

I have attached a quickstart guide I wrote on how to get Swift working with futuregrid by using the new Swift coaster service scripts. This will require the latest development version of Swift. Instructions on how to download/install are in the document. 

Please let me know if you have any questions, if anything is unclear, or if you run into any problems. Thank you! 

Regards, 
David 

On Tue, May 24, 2011 at 3:18 PM, John Bresnahan < bresnaha at mcs.anl.gov > wrote: 

The GPFS server on the FG cluster hotel died yesterday so I cannot get you your credentials.  I'll get back to you when it is up again.  Once it is back the process for getting the needed access keys is described here: 

https://portal.futuregrid.org/tutorials/nimbus 

On 05/23/2011 05:24 AM, David Kelly wrote: 

Hi John, 

I now have a futuregrid account and am added to a project. I am now trying to get our scripts 
working together. 

I ran into a few problems at first when trying to run the futuregrid scripts. On the first system I 
tried I was getting a traceback. It is possible that the system I was using has older versions of 
some of the needed libraries. Then I tried it on a more system that is more frequently updated - my 
laptop running Ubuntu 10.10.  It needed a newer version of the Python crypto tools installed, so I 
installed that (and the python development libraries) and that part seems fine now. 

I am now up to the point of the install script where it is trying to register keys, but it is 
failing. My guess is that I need to change FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY 
in env.sh. I'm not sure what these should be exactly. Are these the contents of my ssh keys, an ssh 
key and a passphrase, or some other type of security? I've tried a few combinations of different 
things but haven't had much luck yet. 

Thanks! 

Regards, 
David 

Traceback from earlier: 
Installing setuptools.......................done. 
  Complete output from command /autonfs/home/davidk/swift-vm-...ython 
/autonfs/home/davidk/swift-vm-...stall pip: 
  Searching for pip 
Reading http://pypi.python.org/simple/pip/ 
Reading http://pip.openplans.org 
Reading http://www.pip-installer.org 
Best match: pip 1.0.1 
Downloading 
http://pypi.python.org/packages/source/p/pip/pip-1.0.1.tar.gz#md5=28dcc70225e5bf925532abc5b087a94b 
Processing pip-1.0.1.tar.gz 
Running pip-1.0.1/setup.py -q bdist_egg --dist-dir 
/tmp/easy_install-GHsjHX/pip-1.0.1/egg-dist-tmp-rXjQ7L 
Traceback (most recent call last): 
  File "/autonfs/home/davidk/swift-vm-boot/ve/bin/easy_install", line 8, in <module> 
    load_entry_point('setuptools==0.6c11', 'console_scripts', 'easy_install')() 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 1712, in main 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 1700, in with_ei_usage 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 1716, in <lambda> 
  File "/soft/python-2.6.1-r1/lib/python2.6/distutils/core.py", line 152, in setup 
    dist.run_commands() 
  File "/soft/python-2.6.1-r1/lib/python2.6/distutils/dist.py", line 975, in run_commands 
    self.run_command(cmd) 
  File "/soft/python-2.6.1-r1/lib/python2.6/distutils/dist.py", line 995, in run_command 
    cmd_obj.run() 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 211, in run 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 446, in easy_install 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 476, in install_item 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 655, in install_eggs 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 930, in build_and_install 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py", 
line 919, in run_setup 
  File 
"/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/sandbox.py", 
line 52, in run_setup 
AttributeError: 'module' object has no attribute '__getstate__' 
---------------------------------------- 
Traceback (most recent call last): 
  File "bin/virtualenv.py", line 1647, in <module> 
    main() 
  File "bin/virtualenv.py", line 558, in main 
    prompt=options.prompt) 
  File "bin/virtualenv.py", line 656, in create_environment 
    install_pip(py_executable) 
  File "bin/virtualenv.py", line 415, in install_pip 
    filter_stdout=_filter_setup) 
  File "bin/virtualenv.py", line 624, in call_subprocess 
    % (cmd_desc, proc.returncode)) 
OSError: Command /autonfs/home/davidk/swift-vm-...ython /autonfs/home/davidk/swift-vm-...stall pip 
failed with error code 1 
Failed to created the needed python virtual environment 

On Fri, May 20, 2011 at 7:20 PM, John Bresnahan < bresnaha at mcs.anl.gov <mailto: bresnaha at mcs.anl.gov >> 

wrote: 

   Our phone call today left me motiviated to show you guys how easy it is to get virtual machines 
   for use with swift on FutureGrid. 

   I made some small scripts around the Nimbus tool cloudinitd.  The scripts just make installing 
   the software and running it trivial.  With a single command you can get N VMs from the 
   FutureGrid Nimbus clouds (N can be on the order of hundreds).  When the tool is done it outputs 
   a line separated list of hostnames.  All of these hostnames have root access available via your 
   ~/.ssh/id_rsa keys. 

   If/when you have FutureGrid credentials, untar the attachment and give it a try.  There are a 
   few minor configurations needed: 

   1) edit the file env.sh and set your FutureGrid security credentials: 

   % cat env.sh 
   export FUTUREGRID_IAAS_ACCESS_KEY=XXXXXXXXXXXXXXXXXX 
   export FUTUREGRID_IAAS_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

   export FUTUREGRID_HOTEL_NODES=2 
   export FUTUREGRID_SIERRA_NODES=2 

   You can also change the value '2' to be whatever number of VMs you want. 

   2) install it on your system.  (this single command downloads and installs everything you need 
   under the cwd): 

   % ./install.sh 

   3) boot the VMs 
   % ./bin/bootit.sh. 
You will see much status output, but the last several lines will be the hostnames acquired from 
   the cloud. 

   Let me know when you guys are ready to check this out! 