[Swift-devel] Re: Getting VMs from FG for use with swift
Michael Wilde
wilde at mcs.anl.gov
Wed Jun 8 09:51:36 CDT 2011
Hi John, David,
A meeting sounds good. When are you in town? Or we can meet by phone tomorrow or Friday.
Re Demos, we can pick from these to start:
- a simple BLAST or "mock BLAST" demo: split a file, blast each segment against a DB,
- protein-RNA docking with "modFTDock" (simple script, real science)
- MODIS: analyze land use satellite data (has graphics)
Mike
----- Original Message -----
> That sounds great. Perhaps we should have another meeting to discuss
> what we can demo from here.
>
> On 06/08/2011 02:38 AM, David Kelly wrote:
> > It's not called directly from Swift, but the start-coaster-service
> > script does that. It calls
> > bootit.sh(cloudinitd), extracts information from the JSON output,
> > sets up reverse SSH tunneling if
> > needed, starts the worker perl script on each node, and then
> > generates Swift configuration files
> > with the correct values.
> >
> > David
> >
> > On Tue, Jun 7, 2011 at 11:27 PM, John Bresnahan
> > <bresnaha at mcs.anl.gov <mailto:bresnaha at mcs.anl.gov>>
> > wrote:
> >
> > Are we at a point where we can have cloudinit.d called from
> > swift and have the hostnames parsed
> > from the json output file? That might make a good demo.
> >
> >
> > On 06/07/2011 03:01 PM, David Kelly wrote:
> >
> > Yep, I am using the VMs I get from cloudinitd with Swift.
> > The only problem I sometimes notice is
> > that cloudinitd can be pretty slow to finish.. somewhere
> > between the point where it prints the
> > information about the hosts (hostname, instance id, etc) but
> > before it prints the success
> > message
> > and exits. I was thinking maybe it's related to the hotel
> > filesystem problems from last
> > week, but
> > I'm not sure. Other times cloudinitd finishes quickly
> > without any major delays.
> >
> > Once it gets past initializing the VMs it runs great. I've
> > been testing with a script called
> > hostsn.swift which basically just calls 'hostname' several
> > times and sends the output to a file.
> > It's useful for verifying that all the VMs are processing
> > work. When you get a chance to test it
> > out, the hostsn script is in the swift examples directory.
> > You can call it with something
> > like this:
> >
> > swift -sites.file sites.xml -tc.file tc.data -config cf
> > hostsn.swift -n=100
> >
> > N is the number of 'hostname' processes to launch.. the
> > files will be created in a directory
> > called
> > outdir.
> >
> > Feel free to send me an email when you get a chance to look
> > at it and we can talk more about it.
> >
> > David
> >
> > On Tue, Jun 7, 2011 at 6:59 PM, John Bresnahan
> > <bresnaha at mcs.anl.gov
> > <mailto:bresnaha at mcs.anl.gov> <mailto:bresnaha at mcs.anl.gov
> > <mailto:bresnaha at mcs.anl.gov>>>
> > wrote:
> >
> > great! I will take a look at that when I can (I am on
> > travel right now).
> >
> > Getting back to using cloudinit.d for swift... we you
> > able to use the VMs you got from
> > FutureGrid with swift? I was hoping we could start
> > encorperating that into swift, or at
> > least
> > acquire a set of VMs and use those as you currently use
> > static machines in some demos.
> >
> > ----- Original Message -----
> > From: "David Kelly" <davidkelly999 at gmail.com
> > <mailto:davidkelly999 at gmail.com>
> > <mailto:davidkelly999 at gmail.com
> > <mailto:davidkelly999 at gmail.com>>>
> > To: "John Bresnahan" <bresnaha at mcs.anl.gov
> > <mailto:bresnaha at mcs.anl.gov>
> > <mailto:bresnaha at mcs.anl.gov <mailto:bresnaha at mcs.anl.gov>>>
> > Cc: "Mike Wilde" <wilde at mcs.anl.gov
> > <mailto:wilde at mcs.anl.gov> <mailto:wilde at mcs.anl.gov
> > <mailto:wilde at mcs.anl.gov>>>, "swift-devel"
> > <swift-devel at ci.uchicago.edu
> > <mailto:swift-devel at ci.uchicago.edu>
> > <mailto:swift-devel at ci.uchicago.edu
> > <mailto:swift-devel at ci.uchicago.edu>>>
> >
> > Sent: Tuesday, June 7, 2011 2:39:20 PM
> > Subject: Re: Getting VMs from FG for use with swift
> >
> > Hello John,
> >
> > I have attached a quickstart guide I wrote on how to get
> > Swift working with futuregrid
> > by using
> > the new Swift coaster service scripts. This will require
> > the latest development version of
> > Swift. Instructions on how to download/install are in
> > the document.
> >
> > Please let me know if you have any questions, if
> > anything is unclear, or if you run into any
> > problems. Thank you!
> >
> > Regards,
> > David
> >
> >
> > On Tue, May 24, 2011 at 3:18 PM, John Bresnahan <
> > bresnaha at mcs.anl.gov
> > <mailto:bresnaha at mcs.anl.gov>
> > <mailto:bresnaha at mcs.anl.gov <mailto:bresnaha at mcs.anl.gov>>
> > > wrote:
> >
> >
> > The GPFS server on the FG cluster hotel died yesterday
> > so I cannot get you your credentials.
> > I'll get back to you when it is up again. Once it is
> > back the process for getting the
> > needed
> > access keys is described here:
> >
> > https://portal.futuregrid.org/tutorials/nimbus
> >
> >
> >
> >
> > On 05/23/2011 05:24 AM, David Kelly wrote:
> >
> >
> >
> >
> >
> > Hi John,
> >
> > I now have a futuregrid account and am added to a
> > project. I am now trying to get our
> > scripts
> > working together.
> >
> > I ran into a few problems at first when trying to run
> > the futuregrid scripts. On the
> > first system I
> > tried I was getting a traceback. It is possible that the
> > system I was using has older
> > versions of
> > some of the needed libraries. Then I tried it on a more
> > system that is more frequently
> > updated - my
> > laptop running Ubuntu 10.10. It needed a newer version
> > of the Python crypto tools
> > installed, so I
> > installed that (and the python development libraries)
> > and that part seems fine now.
> >
> > I am now up to the point of the install script where it
> > is trying to register keys, but
> > it is
> > failing. My guess is that I need to change
> > FUTUREGRID_IAAS_ACCESS_KEY and
> > FUTUREGRID_IAAS_SECRET_KEY
> > in env.sh. I'm not sure what these should be exactly.
> > Are these the contents of my ssh
> > keys, an ssh
> > key and a passphrase, or some other type of security?
> > I've tried a few combinations of
> > different
> > things but haven't had much luck yet.
> >
> > Thanks!
> >
> > Regards,
> > David
> >
> >
> > Traceback from earlier:
> > Installing setuptools.......................done.
> > Complete output from command
> > /autonfs/home/davidk/swift-vm-...ython
> > /autonfs/home/davidk/swift-vm-...stall pip:
> > Searching for pip
> > Reading http://pypi.python.org/simple/pip/
> > Reading http://pip.openplans.org
> > Reading http://www.pip-installer.org
> > Best match: pip 1.0.1
> > Downloading
> > http://pypi.python.org/packages/source/p/pip/pip-1.0.1.tar.gz#md5=28dcc70225e5bf925532abc5b087a94b
> > Processing pip-1.0.1.tar.gz
> > Running pip-1.0.1/setup.py -q bdist_egg --dist-dir
> > /tmp/easy_install-GHsjHX/pip-1.0.1/egg-dist-tmp-rXjQ7L
> > Traceback (most recent call last):
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/bin/easy_install",
> > line 8, in <module>
> > load_entry_point('setuptools==0.6c11',
> > 'console_scripts', 'easy_install')()
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 1712, in main
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 1700, in with_ei_usage
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 1716, in <lambda>
> > File
> > "/soft/python-2.6.1-r1/lib/python2.6/distutils/core.py",
> > line 152, in setup
> > dist.run_commands()
> > File
> > "/soft/python-2.6.1-r1/lib/python2.6/distutils/dist.py",
> > line 975, in run_commands
> > self.run_command(cmd)
> > File
> > "/soft/python-2.6.1-r1/lib/python2.6/distutils/dist.py",
> > line 995, in run_command
> > cmd_obj.run()
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 211, in run
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 446, in easy_install
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 476, in install_item
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 655, in install_eggs
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 930, in build_and_install
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py",
> > line 919, in run_setup
> > File
> > "/autonfs/home/davidk/swift-vm-boot/ve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/sandbox.py",
> > line 52, in run_setup
> > AttributeError: 'module' object has no attribute
> > '__getstate__'
> > ----------------------------------------
> > Traceback (most recent call last):
> > File "bin/virtualenv.py", line 1647, in <module>
> > main()
> > File "bin/virtualenv.py", line 558, in main
> > prompt=options.prompt)
> > File "bin/virtualenv.py", line 656, in
> > create_environment
> > install_pip(py_executable)
> > File "bin/virtualenv.py", line 415, in install_pip
> > filter_stdout=_filter_setup)
> > File "bin/virtualenv.py", line 624, in
> > call_subprocess
> > % (cmd_desc, proc.returncode))
> > OSError: Command /autonfs/home/davidk/swift-vm-...ython
> > /autonfs/home/davidk/swift-vm-...stall pip
> > failed with error code 1
> > Failed to created the needed python virtual environment
> >
> > On Fri, May 20, 2011 at 7:20 PM, John Bresnahan <
> > bresnaha at mcs.anl.gov
> > <mailto:bresnaha at mcs.anl.gov>
> > <mailto:bresnaha at mcs.anl.gov <mailto:bresnaha at mcs.anl.gov>>
> > <mailto: bresnaha at mcs.anl.gov
> > <mailto:bresnaha at mcs.anl.gov> <mailto:bresnaha at mcs.anl.gov
> > <mailto:bresnaha at mcs.anl.gov>> >>
> >
> >
> >
> >
> > wrote:
> >
> > Our phone call today left me motiviated to show you
> > guys how easy it is to get
> > virtual machines
> > for use with swift on FutureGrid.
> >
> > I made some small scripts around the Nimbus tool
> > cloudinitd. The scripts just make
> > installing
> > the software and running it trivial. With a single
> > command you can get N VMs from the
> > FutureGrid Nimbus clouds (N can be on the order of
> > hundreds). When the tool is done
> > it outputs
> > a line separated list of hostnames. All of these
> > hostnames have root access
> > available via your
> > ~/.ssh/id_rsa keys.
> >
> > If/when you have FutureGrid credentials, untar the
> > attachment and give it a try.
> > There are a
> > few minor configurations needed:
> >
> >
> > 1) edit the file env.sh and set your FutureGrid
> > security credentials:
> >
> > % cat env.sh
> > export FUTUREGRID_IAAS_ACCESS_KEY=XXXXXXXXXXXXXXXXXX
> > export
> > FUTUREGRID_IAAS_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> >
> > export FUTUREGRID_HOTEL_NODES=2
> > export FUTUREGRID_SIERRA_NODES=2
> >
> > You can also change the value '2' to be whatever
> > number of VMs you want.
> >
> >
> > 2) install it on your system. (this single command
> > downloads and installs
> > everything you need
> > under the cwd):
> >
> > % ./install.sh
> >
> > 3) boot the VMs
> > % ./bin/bootit.sh.
> > You will see much status output, but the last several
> > lines will be the hostnames
> > acquired from
> > the cloud.
> >
> > Let me know when you guys are ready to check this
> > out!
> >
> >
> >
> >
> >
> >
> >
--
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory
More information about the Swift-devel
mailing list