1.     Requirements

Ruby

https://www.ruby-lang.org/en/

VirtualBox

https://www.virtualbox.org/

EC2 tools (Optional)

http://juanvicenteherrera.eu/2012/02/21/first-steps-with-ec2-api-tools/

Knife

https://learnchef.opscode.com/quickstart/workstation-setup/

Vagrant

$ gem install vagrant

Vagrant plugins

https://github.com/mitchellh/vagrant-aws

https://github.com/schisamo/vagrant-omnibus

https://github.com/cassianoleal/vagrant-butcher

$ vagrant plugin install vagrant-aws
$ vagrant plugin install vagrant-omnibus
$ vagrant plugin install vagrant-butcher

2.     Tests

Vagrant first steps in workstation with VirtualBox

$ vagrant box add base http://files.vagrantup.com/lucid32.box
$ vagrant init
$ vagrant up

Vagrant  test  with manual chef bootstrap

$  vagrant box add CentOS-6.4 http://developer.nrel.gov/downloads/vagrant-boxes/CentOS-6.4-x86_64-v20130309.box
$  vagrant init opscode-ubuntu-1204 https://opscode-vm-bento.s3.amazonaws.com/vagrant/opscode_ubuntu-12.04-i386_chef-11.4.4.box --no-color
$  vagrant up --no-color
$  knife bootstrap localhost  --ssh-user ec2-user   --ssh-password vagrant   --ssh-port 2222   --run-list "recipe[apache2]"   --sudo
$  vagrant ssh

  Vagrant ec2 test 2 without chef management

$ vagrant plugin install vagrant-aws
$ vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
$ vagrant up --provider=aws

3.    POC Vagrant/EC2/Chef

Briefing

Base Box: The base box is simply a saved hard-disk of a Virtual Machine created with VirtualBox. It can contain anything but it needs at least :

• Ruby
• VirtualBox guest additions
• Puppet or Chef

In  the case that use AWS provider the box format is basically just the required metadata.json file along with a Vagrantfile that does default settings for the provider-specific configuration for this provider.

Box used in the example

Vagrant AWS Example Box

Vagrant providers each require a custom provider-specific box format. These files compose the contents of a box for the aws provider.

• README.md
• Vagrantfile
• metadata.json

To turn this into a box:

$ tar cvzf aws.box ./metadata.json ./Vagrantfile

This box works by using Vagrant’s built-in Vagrantfile merging to setup defaults for AWS. These defaults can easily be overwritten by higher-level Vagrantfiles (such as project root Vagrantfiles).

Vagrant basic commands

vagrant init :

creates a file called ‘Vagrantfile’ in your current directory

when you look at the file, it will contain the directive …

config.vm.box = "base"

this is what makes the link to the box we called ‘base’

you can further edit the Vagrantfile before you start it

vagrant up:

up until now, no virtual machine was created

therefore vagrant will import the disks from the box ‘base’ into Virtualbox(not applied for AWS, the instance that is going to be created has EBS disks already configured)

map via NAT the port 22 from your VM to a free local port

it will create a .vagrant file : a file that contains a mapping between your description ‘base’ and the UUID of the virtual machine

vagrant ssh:

this will lookup the mapping of the ssh inside and will execute the SSH process to log into the machine use a privatekey of use vagrant to login to a box that has the user vagrant with it’s public setup in the virtual machine

Vagrant commands

$ vagrant
Tasks:
vagrant box                        # Commands to manage system boxes
vagrant destroy                    # Destroy the environment, deleting the created virtual machines
vagrant halt                       # Halt the running VMs in the environment
vagrant help [TASK]                # Describe available tasks or one specific task
vagrant init [box_name] [box_url]  # Initializes the current folder for Vagrant usage
vagrant package                    # Package a Vagrant environment for distribution
vagrant provision                  # Rerun the provisioning scripts on a running VM
vagrant reload                     # Reload the environment, halting it then restarting it.
vagrant resume                     # Resume a suspended Vagrant environment.
vagrant ssh                        # SSH into the currently running Vagrant environment.
vagrant ssh_config                 # outputs .ssh/config valid syntax for connecting to this environment via ssh
vagrant status                     # Shows the status of the current Vagrant environment.
vagrant suspend                    # Suspend a running Vagrant environment.
vagrant up                         # Creates the Vagrant environment
vagrant version                    # Prints the Vagrant version information

POC

Add dummy AWS box

$ vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

Vagrant example with 1 ec2 instance and configuration managed by Chef

Vagrantfile

[sourcecode language=»ruby»]

Vagrant.configure("2") do |config|

config.vm.box = "dummy"

config.omnibus.chef_version = "11.6.0"

config.vm.provider :aws do |aws, override|

aws.keypair_name = "bq"

aws.access_key_id     = " xxxxxx "

aws.secret_access_key = " xxxxxxxxxxxxxxxxxx "

aws.ami = "ami-3ad1af53"

override.ssh.username = "ec2-user"

override.ssh.private_key_path = "/Users/juanvi/keypairs/bq.pem"

end

config.vm.provision :chef_client do |chef|

chef.chef_server_url = "https://api.opscode.com/organizations/juanvi"

chef.validation_key_path = "/Users/juanvi/chef-repo/.chef/juanvi-validator-new.pem"

chef.validation_client_name = "juanvi-validator"

# Provision with the database role

chef.add_role("webserver")

# Set the environment for the chef server

chef.environment = "dev"

end

end

[/sourcecode]

Notes: The ami used as template is a standard AMI 64 bits EBS backed http://aws.amazon.com/amazon-linux-ami/ with sudo permissions for vagrant user.

Vagrant example with 2 instances in ec2 and configuration managed by Chef

[sourcecode language=»ruby»]

Vagrant.configure("2") do |config|

config.vm.define :web do |web|

web.vm.box = "dummy"

web.vm.provider :aws do |aws, override|

aws.keypair_name = "bq"

config.omnibus.chef_version = "11.6.0"

aws.access_key_id     = " xxxxxx "

aws.secret_access_key = " xxxxxxxxxxxxxxxxxx "

aws.ami = "ami-3ad1af53"

override.ssh.username = "ec2-user"

override.ssh.private_key_path = "/Users/juanvi/keypairs/bq.pem"

end

web.vm.provision :chef_client do |chef|

chef.chef_server_url = "https://api.opscode.com/organizations/juanvi"

chef.validation_key_path = "/Users/juanvi/chef-repo/.chef/juanvi-validator-new.pem"

chef.validation_client_name = "juanvi-validator"

#  Provision with the database role

chef.add_role("webserver")

# Set the environment for the chef server

chef.environment = "dev"

end

end

config.vm.define :db do |db|

db.vm.box = "dummy"

db.vm.provider :aws do |aws, override|

aws.keypair_name = "bq"

config.omnibus.chef_version = "11.6.0"

aws.access_key_id     = " xxxxxx "

aws.secret_access_key = " xxxxxx "

aws.ami = "ami-3ad1af53"

override.ssh.username = "ec2-user"

override.ssh.private_key_path = "/Users/juanvi/keypairs/bq.pem"

end

db.vm.provision :chef_client do |chef|

chef.chef_server_url = "https://api.opscode.com/organizations/juanvi"

chef.validation_key_path = "/Users/juanvi/chef-repo/.chef/juanvi-validator-new.pem"

chef.validation_client_name = "juanvi-validator"

#  Provision with the database role

chef.add_role("db")

# Set the environment for the chef server

chef.environment = "dev"

end

end

end
[/sourcecode]

Provisioning only one kind of server executing chef_client

$ vagrant provision web --provision-with chef_client

Provisioning the whole platform executing chef_client

$ vagrant provision --provision-with chef_client

After editing the Vagrantfile you need to ‘reboot’ the machine to take this settings

$ vagrant reload

Version control

Now is a good time to version control your vagrant project

$ cd
$ git init
$ git add Vagrantfile
$ git commit -m "First version of Vagranfile"

Cleanup

Install the following plugin for vagrant:

$ vagrant plugin install vagrant-butcher

add this line to your Vagrantfile:

[sourcecode language=»ruby»]

config.butcher.knife_config_file = ‘.chef/knife.rb’

[/sourcecode]

then you can terminate an instance and deregister in the Chef server executing:

$ vagrant destroy -f
[Butcher] knife.rb location set to '/path/to/knife.rb'
[Butcher] Chef node 'node_name' successfully butchered from the server...
[Butcher] Chef client 'node_name' successfully butchered from the server...
[default] Forcing shutdown of VM...
[default] Destroying VM and associated drives...

4.    Tips

• Enabling different settings based on environment

Setting these variables is as easy as prepending them to the vagrant command

$ vagrant_env=development vagrant up

• Create a config file in ruby with the aws credentials and include in all of the vagrant files to have only one configuration file:

[sourcecode language=»ruby»]

require File.expand_path(‘~/.chef/aws_credentials.rb’)

[/sourcecode]

inside Vagrantfile you can use these variables with:

[sourcecode language=»ruby»]

access_key_id=<strong>@access_key_id</strong>

[/sourcecode]

• I recommend you have all the chef certificates in your ~/.chef folder  and a knife.rb file per project with specific configuration(path to chef certificates and ec2 keypairs in your ~/.chef folder)
• If you include .vagrant folder in your git repositor all the team can manage the vagrant environment
• Create a base chef role with all of the common tools needed in all of the servers (basic tools)

[sourcecode language=»ruby»]
{
"name": "web-base",
"description": "Base role applied to all nodes.",
"run_list":[
"recipe[git]",
"recipe[build-essential]",
"recipe[vim]",
"recipe[aws_hostname]",
"recipe[aws_hostname::register-dns]"
],
"override_attributes":{},
"chef_type":"role"

}
[/sourcecode]

5.     Some  observations

• It clearly helps everybody to have a consistent environment to develop against, the latest version is just one git pull away.
• The central approach drives people to a) do frequent commits and b) do stable commits.
• The task of writing recipes is not picked up by all team members, and seem to stay the main job of the system oriented people on the team.
• Reading coobooks help people understand what is needed and makes it easy to point out what needs to be changed. But learning the skills to write recipes/manifest is a blocking factor just as having a backend developer writing frontend code.
• The test the admins have to do before committing their cookbooks, is that they destroy their own ‘development’ box and re-provision a new box to see if this works.
• The longer the provision takes, the less frequent people do it. It’s important to keep that process as fast as possible. It’s all about feedback and we want it fast.
• Installation problems would get noticed far sooner in the process.

People would only do a full rebuild in the morning when getting their coffee.

6.     Conclusions

• Vagrant is an essential part of the DevOps  process: it is the solution to developing, testing and deploying in the same environment. It thus ensures a smoother transition of your project from the dev team to the ops team.
• Vagrant is EASY. And it is compatible with Chef.

7.     References and resources

You can check the cookbook used in these examples and all of the chef resources ( https://github.com/juanviz/chef-jv/blob/master/cookbooks/wiki_app/README.md and https://github.com/juanviz/chef-jv/) and more Vagrant examples (https://github.com/juanviz/VagrantProjects)

http://docs.vagrantup.com/v2/cli/index.html

http://red-badger.com/blog/2013/02/21/automating-your-infrastructure-with-vagrant-chef-from-development-to-the-cloud/

http://docs-v1.vagrantup.com/v1/docs/provisioners/chef_server.html

http://www.jasongrimes.org/2012/06/managing-lamp-environments-with-chef-vagrant-and-ec2-1-of-3/

http://docs-v1.vagrantup.com/v1/docs/getting-started/provisioning.html

http://beacon.wharton.upenn.edu/404/2013/04/starting-the-server-build/