hadooppowered.com

Create a Debian Wheezy Vagrant box

Intro

For hacking around with Hadoop stuff when you don't have a dev-cluster at hand, on a train, plane or in the park with bad a internet connection, you can easily use Vagrant boxes on your computer.

Vagrant describes itself as:

Create and configure lightweight, reproducible, and portable development environments.

This tutorial will show you how to setup a minimalistic Vagrant box for further playing around.

This tutorial will not show how to install Hadoop in a Vagrant box.

Prerequisites

To be able to follow the steps below you need some things first.

Create a new virtual machine

Create a new virtual machine with virtual and name it "vagrant-debian-wheezy".

Note: Disable audio in the settings.

Choose "Expert Mode" as setup method and use the default settings if not stated otherwise below.

Configure the virtual machine

Login to your new virtual machine as root an do the following steps:

Update your system and install a few basic tools
$ apt-get update
$ apt-get upgrade
$ apt-get install -y build-essential module-assistant
$ module-assistant prepare
$ apt-get install -y sudo zerofree openssh-server
Grant the user vagrant full sudo rights
$ visudo
vagrant ALL=(ALL) NOPASSWD: ALL

$ /etc/init.d/sudo restart
Install the VirtualBox guest extensions

You have to mount the Virtualbox guest extensions iso image in the VBox Manager first!

$ mount /media/cdrom
$ sh /media/cdrom/VBoxLinuxAdditions.run
$ sudo ln -s /opt/VBoxGuestAdditions-*/lib/VBoxGuestAdditions /usr/lib/VBoxGuestAdditions

Shutdown the machine when finished.

SSH configuration

Enable local ssh access to your virtual machine (vm must be shut down!):

$ VBoxManage modifyvm "vagrant-debian-wheezy" --natpf1 "ssh,tcp,,2222,,22"

Generate a new ssh keypair without a passphrase for use with your virtual machine and add it to your local ssh-agent:

$ ssh-keygen -t rsa -b 4096 -P '' -f ~/.ssh/vagrant
$ ssh-add ~/.ssh/vagrant

Boot the virtual machine now.

Test if you can connect and create an empty .ssh directory for the user vagrant.

$ ssh -p 2222 vagrant@127.0.0.1
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh

Copy the ssh public key to your vm's authorized_keys file

$ scp -P 2222 ~/.ssh/vagrant.pub vagrant@127.0.0.1:.ssh/authorized_keys

Test the SSH connection with your key

$ ssh -p 2222 -i ~/.ssh/vagrant vagrant@127.0.0.1

# if you used the "ssh-add" command from above you can omit the "-i" option 
$ ssh -p 2222 vagrant@127.0.0.1

Cleanup the machine

You need to do this, to keep the final vagrantfile as small as possible.

$ rm -rf /usr/share/doc
$ rm -rf /usr/src/linux-headers*
$ rm -rf /usr/src/linux
$ find /var/cache -type f -exec rm -rf {} \;

Remove unused locales (keep your locales by adding ! -name "YOUR LOCALE" to the command below)

$ find /usr/share/locale/* -maxdepth 1 -type d ! -name "de*" ! -name "en*" -exec rm -rf {} \;

Remove virtual box guest extension sources

$ rm -rf /usr/src/vboxguest*
$ rm -rf /usr/src/virtualbox-ose-guest*

Zerofree

zerofree finds the unallocated, blocks with non-zero value content in an ext2, ext3 or ext4 filesystem (e.g. /dev/sda1) and fills them with zeroes. This makes the disk easier to compress for Vagrant.

Login as root via VirtualBox. This will not not work when SSH'd into the machine because the command init 1 takes the system into single user mode. This means that services like sshd will be terminated.

$ init 1
$ mount -o remount,ro /dev/sda1
$ zerofree /dev/sda1 # this will take a while...
$ shutdown -h now

Don't forget to shutdown the virtual machine when done!

Boxing the virtual machine

Create a directory where to store the vagrant box (I used tmp/src/vagrant):

$ mkdir -p ~/tmp/vagrant/src
$ cd ~/tmp/vagrant/src

Box the machine!

$ vagrant package --base vagrant-debian-wheezy
==> vagrant-debian-wheezy: Clearing any previously set forwarded ports...
==> vagrant-debian-wheezy: Exporting VM...
==> vagrant-debian-wheezy: Compressing package to: ~/tmp/src/vagrant/package.box

Add it to your repository

$ vagrant box add vagrant-debian-wheezy package.box
==> box: Adding box 'vagrant-debian-wheezy' (v0) for provider:
    box: Downloading: file:///Users/alex/tmp/src/vagrant/package.box
==> box: Successfully added box 'vagrant-debian-wheezy' (v0) for 'virtualbox'!

Ensure it is added

$ vagrant box list
vagrant-debian-wheezy (virtualbox, 0)

Testing the new box

Create a directory for your vagrant file

$ mkdir -p ~/tmp/vagrant/env
$ cd ~/tmp/vagrant/env

Initialize your vagrant box

$ vagrant init vagrant-debian-wheezy
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

Cleanup, add SSH-Agent forwarding and your ssh key to the vagrantfile

$ sed -i ".bak" -E '/(^[ ]+#.*$)|^$/d' Vagrantfile # remove comments and empty lines, make a backup

$ vi Vagrantfile
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
...
 config.ssh.forward_agent = true
 config.ssh.private_key_path = "~/.ssh/vagrant"
...

Start the box and ssh into it

$ vagrant up
$ vagrant ssh

# or use the old fashioned way...
$ ssh -i ~/.ssh/vagrant -p 2222 vagrant@127.0.0.1

When done, you can "reset" the vagrant box with vagrant destroy. This will turn back the box to the default configuration you just set up during this tutorial. Everything you installed or configured in this machine after the vagrant up command will be lost forever.

$ vagrant destroy
    default: Are you sure you want to destroy the 'default' VM? [y/N] y
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...

This completes the tutorial. I hope everything was clear enough and was working when you tried to set up (your first) vagrant box.

Have fun,

Alex


No public comments, but you can email comments@hadooppowered.com or tweet @hadooppowered.


← Older
Newer →