The ‘name’ attribute is required in cookbook metadata: Solving a Vagrant/Chef Provisioning Issue

2015-04-21T00:00:00+00:00

When Vagrant/Chef Provisioning Goes South

I recently ran into the following error when provisioning a new vagrant machine via the vagrant up command:

[2015-04-21T17:10:35+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-04-21T17:10:35+00:00] ERROR: Cookbook loaded at path(s) [/tmp/vagrant-chef/path/to/my-cookbook] has invalid metadata: The 'name' attribute is required in cookbook metadata
[2015-04-21T17:10:35+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

After some googling and digging I learned version 12 of chef-client introduced a breaking change. From version 12 on, every cookbook requires a name attribute in their metadata.rb file. A quick grep through the metadata.rb files in the project revealed several did not include name attributes. You would be correct at this point to suggest I could have added name attributes to the cookbook metadata files and been done with this. However, in this case I was a consumer of the cookbooks and was not involved in maintaining them so an alternate solution was preferable.

Selecting a Specific Version of Chef in Vagrant

My idea for a solution was to install the most recent chef-client release prior to version 12. I was not sure how to do this initially but along the way I learned that by default, Vagrant will install the most recent release of chef-client. The Vagrant documentation for Chef provisioners described what I needed to do. The Chef version could be specified in config.vm.provision block in the Vagrantfile:

config.vm.provision :chef_solo do |chef|
      chef.version = "11.18"
      chef.cookbooks_path = "cookbooks"
      chef.data_bags_path = "data_bags"

      # List of recipes to run
      chef.add_recipe "vagrant_main::my_project"
  end

With this configuration change, chef-client 11.18 completed the provisioning step successfully.

Testing your chef repo pull requests with chef-zero, Vagrant and Jenkins

2015-02-18T00:00:00+00:00

All Liquid Galaxy setups deployed by End Point are managed by Chef. Typical deployment consists of approx 3 to 9 Linux boxes from which only 1 is managed and the rest is an ISO booted from this machine via network with copy-on-write root filesystem. Because of this, typical deployment involves more steps than just updating your code and restarting application. Deployment + rollback may be even 10 times longer compared with typical web application. Due to this fact, we need to test our infrastructure extensively.

What are we to do in order to make sure that our infrastructure is tested well before it hits production?

Scary? It’s not.

Workflow broken down by pieces

lg_chef.git repo—where we keep cookbooks, environments and node definitions
GitHub pull request—artifact of infrastructure source code tested by Jenkins
Vagrant—virtual environment in which chef is run in order to test the artifact. There’s always 1 master node and few Vagrant boxes that boot an ISO from master via tftp protocol
chef-zero—Chef flavor used to converge and test the infrastructure on the basis of GitHub pull request
chef-server/chef-client—Chef flavor used to converge and test production and pre-production environment
Jenkins — Continuous Integration environment that runs the converge process and part of the tests
Tests—two frameworks used—BATS (for the integration tests on the top) and minitest (for after-converge tests)
lg-live-build—our fork of Debian live build used to build the ISO that is booted by Vagrant slaves

Workflow broken down by the order of actions

User submits GitHub pull request to lg_chef.git repo
GitHub pull request gets picked up by Jenkins
Jenkins creates 1 master Vagrant node and several slave nodes to act as slaves
chef-zero converges master Vagrant box and runs minitest
BATS tests run on the freshly converged Vagrant masterbox. Few steps are performed here: ISO is built, it’s distributed to the slaves, slaves boot the ISO and final integration tests are run to see whether slaves have all the goodness.
If points 1 to 5 are **green,**developer merges the changes, uploads the updated cookbooks, node definitions, roles and environments and runs the final tests.

What didn’t work for us and why

kitchen-vagrant —because it didn’t play well with Jenkins (or JVM itself) and didn’t know how to use advanced Vagrant features for specifying multiple networking options, interfaces and drivers. However it supports using your own Vagrantfile.erb
We’ve had some doubts about keeping all the cookbooks, environments and node definitions in one repo because chef-server/chef-client tests can only test your stuff if it’s uploaded to the Chef server, but chef-zero came in handy

The code

As previously mentioned, we needed our own vagrant template file.

Vagrant.configure("2") do |config|
  <% if  @data[:chef_type] == "chef_zero" %>
  config.chef_zero.enabled = true
  config.chef_zero.chef_server_url = "<%= @data[:chef_server_url] %>"
  config.chef_zero.roles = "../../roles/"
  config.chef_zero.cookbooks = "../../cookbooks/"
  config.chef_zero.environments = "../../environments/"
  config.chef_zero.data_bags = "../integration/default/data_bags/"
  <% else %>
  config.chef_zero.enabled = false
  <% end %>
  config.omnibus.chef_version = "<%= @data[:chef_version] %>"
  config.vm.define "<%= @data[:headnode][:slug] %>" do |h|
  h.vm.box = "<%= @data[:headnode][:box] %>"
    h.vm.box_url = "<%= @data[:headnode][:box_url] %>"
    h.vm.hostname = "<%= @data[:headnode][:hostname] %>"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
      <% @data[:headnode][:customizations].each  do |key, value| %>
        p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
      <% end %>
    end
    h.vm.provision :chef_client do |chef|
      <% if @data[:chef_type] == "chef_zero" %>
      chef.environment = "<%= @data[:headnode][:provision][:environment] %>"
      chef.run_list = <%= @data[:run_list] %>
      chef.json = <%= @data[:node_definition] %>
      chef.chef_server_url = "<%= @data[:chef_server_url] %>"
      <% else %>
      chef.chef_server_url = "<%= @data[:headnode][:provision][:chef_server_url] %>"
      <% end  %>
      chef.validation_key_path = "<%= @data[:headnode][:provision][:validation_key_path] %>"
      chef.encrypted_data_bag_secret_key_path = "<%= @data[:headnode][:provision][:encrypted_data_bag_secret_key_path] %>"
      chef.verbose_logging = <%= @data[:headnode][:provision][:verbose_logging] %>
      chef.log_level = "<%= @data[:headnode][:provision][:log_level] %>"
  <% end  %>
  config.omnibus.chef_version = "<%= @data[:chef_version] %>"
  config.vm.define "<%= @data[:headnode][:slug] %>" do |h|
  h.vm.box = "<%= @data[:headnode][:box] %>"
    h.vm.box_url = "<%= @data[:headnode][:box_url] %>"
    h.vm.hostname = "<%= @data[:headnode][:hostname] %>"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
      <% @data[:headnode][:customizations].each  do |key, value| %>
        p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
      <% end %>
    end
    h.vm.provision :chef_client do |chef|
      <% if @data[:chef_type] == "chef_zero" %>
      chef.environment = "<%= @data[:headnode][:provision][:environment] %>"
      chef.run_list = <%= @data[:run_list] %>
      chef.json = <%= @data[:node_definition] %>
      chef.chef_server_url = "<%= @data[:chef_server_url] %>"
      <% else %>
      chef.chef_server_url = "<%= @data[:headnode][:provision][:chef_server_url] %>"
      <% end  %>
      chef.validation_key_path = "<%= @data[:headnode][:provision][:validation_key_path] %>"
      chef.encrypted_data_bag_secret_key_path = "<%= @data[:headnode][:provision][:encrypted_data_bag_secret_key_path] %>"
      chef.verbose_logging = <%= @data[:headnode][:provision][:verbose_logging] %>
      chef.log_level = "<%= @data[:headnode][:provision][:log_level] %>"
      chef.node_name = "<%= @data[:headnode][:provision][:node_name] %>"
    end
  end

  #display nodes
  <% @data[:display_nodes][:nodes].each  do |dn| %>
  config.vm.define "<%= dn[:slug] %>" do |dn_config|
    dn_config.vm.box_url = "<%= @data[:display_nodes][:global][:box_url] %>"
    dn_config.vm.hostname = "<%= dn[:hostname] %>"
    dn_config.vm.box = "<%= @data[:display_nodes][:global][:box] %>"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
    <% @data[:display_nodes][:global][:customizations].each  do |key, value| %>
      p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
    <% end %>
      p.customize ["modifyvm", :id, "--macaddress1", "<%= dn[:mac] %>"]
      p.customize ["createhd", "--filename", "../files/<%= dn[:slug] %>.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/<%= dn[:slug] %>.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_<%= dn[:slug] %>.vmdk"]
    end
  end
  <% end %>
end

It renders a Vagrant File out of following data:

{
  "description" : "This file is used to generate Vagrantfile and run_test.sh and also run tests. It should contain _all_ data needed to render the templates and run teh tests.",
  "chef_version" : "11.12.4",
  "chef_type" : "chef_zero",
  "vagrant_template_file" : "vagrantfile.erb",
  "run_tests_template_file" : "run_tests.sh.erb",
  "chef_server_url" : "http://192.168.1.2:4000",
  "headnode" :
    {
    "slug" : "projectX-pull-requests",
    "box" : "opscode-ubuntu-14.04",
    "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
    "hostname" : "lg-head",
    "bats_tests_dir" : "projectX-pr",
    "customizations" : {
      "--memory" : "2048",
      "--cpus": "2",
      "--nic1" : "nat",
      "--nic2": "intnet",
      "--nic3": "none",
      "--nic4" : "none",
      "--nictype1": "Am79C970A",
      "--nictype2": "Am79C970A",
      "--intnet2": "projectX-pull-requests"
    },
    "provision" : {
      "chef_server_url" : "https://chefserver.ourdomain.com:40443",
      "validation_key_path" : "~/.chef/validation.pem",
      "encrypted_data_bag_secret_key_path" : "~/.chef/encrypted_data_bag_secret",
      "node_name" : "lg-head-projectXtest.liquid.glx",
      "environment" : "pull_requests",
      "verbose_logging" : true,
      "log_level" : "info"
    }
  },
  "display_nodes" : {
    "global" : {
      "box" : "opscode-ubuntu-14.04",
      "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
      "customizations" : {
        "--memory" : "2048",
        "--cpus" : "1",
        "--boot1" : "floppy",
        "--boot2" : "net",
        "--boot3" : "none",
        "--boot4" : "none"
    },
    "provision" : {
      "chef_server_url" : "https://chefserver.ourdomain.com:40443",
      "validation_key_path" : "~/.chef/validation.pem",
      "encrypted_data_bag_secret_key_path" : "~/.chef/encrypted_data_bag_secret",
      "node_name" : "lg-head-projectXtest.liquid.glx",
      "environment" : "pull_requests",
      "verbose_logging" : true,
      "log_level" : "info"
    }
  },
  "display_nodes" : {
    "global" : {
      "box" : "opscode-ubuntu-14.04",
      "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
      "customizations" : {
        "--memory" : "2048",
        "--cpus" : "1",
        "--boot1" : "floppy",
        "--boot2" : "net",
        "--boot3" : "none",
        "--boot4" : "none",
        "--intnet1" : "projectX-pull-requests",
        "--nicpromisc1": "allow-all",
        "--nic1" : "intnet",
        "--nic2": "none",
        "--nic3": "none",
        "--nic4": "none",
        "--nictype1": "Am79C970A",
        "--ioapic": "on"
      }
    },
    "nodes" : [
      {
      "slug" : "projectX-pull-requests-kiosk",
      "hostname" : "kiosk",
      "mac" : "5ca1ab1e0001"
    },
    {
      "slug" : "projectX-pull-requests-display",
      "hostname" : "display",
      "mac" : "5ca1ab1e0002"
    }
    ]
  }
}

As a result we get an on-the-fly Vagrantfile that’s used during the testing:

Vagrant.configure("2") do |config|

  config.chef_zero.enabled = true
  config.chef_zero.chef_server_url = "http://192.168.1.2:4000"
  config.chef_zero.roles = "../../roles/"
  config.chef_zero.cookbooks = "../../cookbooks/"
  config.chef_zero.environments = "../../environments/"
  config.chef_zero.data_bags = "../integration/default/data_bags/"

  config.omnibus.chef_version = "11.12.4"
  config.vm.define "projectX-pull-requests" do |h|
  h.vm.box = "opscode-ubuntu-14.04"
    h.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    h.vm.hostname = "lg-head"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
        p.customize ["modifyvm", :id, "--memory", "2048"]
        p.customize ["modifyvm", :id, "--cpus", "2"]
        p.customize ["modifyvm", :id, "--nic1", "nat"]
        p.customize ["modifyvm", :id, "--nic2", "intnet"]
        p.customize ["modifyvm", :id, "--nic3", "none"]
        p.customize ["modifyvm", :id, "--nic4", "none"]
        p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
        p.customize ["modifyvm", :id, "--nictype2", "Am79C970A"]
        p.customize ["modifyvm", :id, "--intnet2", "projectX-pull-requests"]
    end
    h.vm.provision :chef_client do |chef|

      chef.environment = "pull_requests"
      chef.run_list = ["role[lg-head-nocms]", "recipe[lg_live_build]", "recipe[lg_tftproot]", "recipe[lg_projectX]", "recipe[lg_test]", "recipe[test_mode::bats]", "recipe[hostsfile::projectX]"]
      chef.json = {:sysctl=>{:params=>{:vm=>{:swappiness=>20}, :net=>{:ipv4=>{:ip_forward=>1}}}}, :test_mode=>true, :call_ep=>{:keyname=>"ProjectX CI Production", :fwdport=>33299}, :tftproot=>{:managed=>true}, :tags=>[], :lg_cms=>{:remote=>"github"}, :monitor=>true, :lg_grub=>{:cmdline=>"nomodeset biosdevname=0"}, :projectX=>{:repo_branch=>"development", :display_host=>"42-a", :kiosk_host=>"42-b", :sensors_host=>"42-b", :maps_url=>"https://www.google.com/maps/@8.135687,-75.0973243,17856994a,40.4y,1.23h/data=!3m1!1e3?esrch=Tactile::TactileAcme,Tactile::ImmersiveModeEnabled"}, :liquid_galaxy=>{:touchscreen_link=>"/dev/input/lg_active_touch", :screenshotd=>{:screen_rows=>"1", :screen_columns=>"1"}, :screenshot_service=>true, :display_nodes=>[{:hostname=>"42-c"}, {:allowed_pages=>["Google Maps", "Pacman Doodle", "jellyfish", "Doodle Selection", "ProjectX Video Player", "Composer kiosk"], :hostname=>"42-b", :mac=>"5c:a1:ab:1e:00:02", :features=>"mandatory_windows, plain_gray, starry_skies", :bad_windows_names=>"Google Earth - Login Status", :mandatory_windows_names=>"awesome", :screens=>[{:display=>":0", :crtc=>"default", :grid_order=>"0"}], :screen_rotation=>"normal", :audio_device=>"{type hw; card DGX; device 0}", :onboard_enable=>true, :keyboard_enable=>true, :mouse_enable=>true, :cursor_enable=>true, :background_extension=>"jpg", :background_mode=>"zoom-fill", :projectX=>{:extensions=>{:kiosk=>"ProjectX Kiosk", :google_properties_menu=>"Google Properties Menu", :onboard=>"Onboard", :no_right_click=>"Right Click Killer", :render_statistics=>"Render Statistics"}, :browser_slug=>"lgS0", :urls=>"https://www.google.com/maps", :ros_nodes=>[{:name=>"rfreceiver_reset", :pkg=>"rfreceiver", :type=>"kill_browser.py"}, {:name=>"proximity", :pkg=>"maxbotix", :type=>"sender.py"}, {:name=>"spacenav", :pkg=>"spacenav_node", :type=>"spacenav_node"}, {:name=>"leap", :pkg=>"leap_motion", :type=>"sender.py"}, {:name=>"projectX_nav", :pkg=>"projectX_nav", :type=>"projectX_nav"}, {:name=>"onboard", :pkg=>"onboard", :type=>"listener.py"}, {:name=>"rosbridge", :pkg=>"rosbridge_server", :type=>"rosbridge_websocket", :params=>[{:name=>"certfile", :value=>"/home/lg/etc/ros.crt"}, {:name=>"keyfile", :value=>"/home/lg/etc/ros.key"}]}]}, :browser_infinite_url=>"http://lg-head/projectX-loader.html"}, {:hostname=>"42-a", :mac=>"5c:a1:ab:1e:00:01", :features=>"mandatory_windows, plain_gray, starry_skies, erroneous_text", :bad_windows_names=>"Google Earth - Login Status", :mandatory_windows_names=>"awesome", :screens=>[{:display=>":0", :crtc=>"default", :grid_order=>"1"}], :keyboard_enable=>true, :mouse_enable=>true, :cursor_enable=>true, :background_extension=>"jpg", :background_mode=>"zoom-fill", :nvidia_mosaic=>true, :manual_layout=>{:default=>"1024x768+0+0"}, :projectX=>{:extensions=>{:display=>"ProjectX Large Display", :pacman=>"pacman", :render_statistics=>"Render Statistics"}, :browser_slug=>"lgS0", :urls=>"https://www.google.com/maps", :ros_nodes=>[{:name=>"geodata", :pkg=>"geodata", :type=>"geodata_server.py"}]}, :browser_infinite_url=>"http://lg-head/projectX-loader.html", :default_browser_bin=>"google-chrome", :allowed_pages=>["Google Maps", "Pacman Doodle", "jellyfish", "ProjectX Video Player", "Composer wall"]}], :has_cec=>false, :google_office=>false, :viewsync_master=>"42-b", :has_touchscreen=>false, :has_spacenav=>true, :support_name=>"projectX-ci", :podium_interface=>"http://lg-head", :podium_display=>"42-b:default"}}
      chef.chef_server_url = "http://192.168.1.2:4000"

      chef.validation_key_path = "~/.chef/validation.pem"
      chef.encrypted_data_bag_secret_key_path = "~/.chef/encrypted_data_bag_secret"
      chef.verbose_logging = true
      chef.log_level = "info"
      chef.node_name = "lg-head-projectXtest.liquid.glx"
    end
  end

  #display nodes

  config.vm.define "projectX-pull-requests-kiosk" do |dn_config|
    dn_config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    dn_config.vm.hostname = "kiosk"
    dn_config.vm.box = "opscode-ubuntu-14.04"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
      p.customize ["modifyvm", :id, "--memory", "2048"]
      p.customize ["modifyvm", :id, "--cpus", "1"]
      p.customize ["modifyvm", :id, "--boot1", "floppy"]
      p.customize ["modifyvm", :id, "--boot2", "net"]
      p.customize ["modifyvm", :id, "--boot3", "none"]
      p.customize ["modifyvm", :id, "--boot4", "none"]
      p.customize ["modifyvm", :id, "--intnet1", "projectX-pull-requests"]
      p.customize ["modifyvm", :id, "--nicpromisc1", "allow-all"]
      p.customize ["modifyvm", :id, "--nic1", "intnet"]
      p.customize ["modifyvm", :id, "--nic2", "none"]
      p.customize ["modifyvm", :id, "--nic3", "none"]
      p.customize ["modifyvm", :id, "--nic4", "none"]
      p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
      p.customize ["modifyvm", :id, "--ioapic", "on"]
      p.customize ["modifyvm", :id, "--macaddress1", "5ca1ab1e0001"]
      p.customize ["createhd", "--filename", "../files/projectX-pull-requests-kiosk.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/projectX-pull-requests-kiosk.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_projectX-pull-requests-kiosk.vmdk"]
    end
  end

  config.vm.define "projectX-pull-requests-display" do |dn_config|
    dn_config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    dn_config.vm.hostname = "display"
    dn_config.vm.box = "opscode-ubuntu-14.04"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
      p.customize ["modifyvm", :id, "--memory", "2048"]
      p.customize ["modifyvm", :id, "--cpus", "1"]
      p.customize ["modifyvm", :id, "--boot1", "floppy"]
      p.customize ["modifyvm", :id, "--boot2", "net"]
      p.customize ["modifyvm", :id, "--boot3", "none"]
      p.customize ["modifyvm", :id, "--boot4", "none"]
      p.customize ["modifyvm", :id, "--intnet1", "projectX-pull-requests"]
      p.customize ["modifyvm", :id, "--nicpromisc1", "allow-all"]
      p.customize ["modifyvm", :id, "--nic1", "intnet"]
      p.customize ["modifyvm", :id, "--nic2", "none"]
      p.customize ["modifyvm", :id, "--nic3", "none"]
      p.customize ["modifyvm", :id, "--nic4", "none"]
      p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
      p.customize ["modifyvm", :id, "--ioapic", "on"]
      p.customize ["modifyvm", :id, "--macaddress1", "5ca1ab1e0002"]
      p.customize ["createhd", "--filename", "../files/projectX-pull-requests-display.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/projectX-pull-requests-display.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_projectX-pull-requests-display.vmdk"]
    end
  end

end

Finally we have the environment stored in one Vagrantfile. The missing part is to how to run tests on it.

The testing script does the following:

#/bin/bash
set -e
# FUNCTIONS

function halt_vm () {
  vms=`vboxmanage list vms | grep "vagrant_$1_" | awk {'print $1'} | sed s/'"'//g`
  echo "Stopping VM $vms"
  stop_result=$(for vm in $vms ; do vboxmanage controlvm $vm poweroff; echo $?; done)
  echo "Output of stopping VM $1 : $stop_result"
}

function boot_vm () {
  vms=`vboxmanage list vms | grep "vagrant_$1_" | awk {'print $1'} | sed s/'"'//g`
  echo "Booting VM $vms"
  start_result=$(for vm in $vms ; do vboxmanage startvm $vm --type headless; echo $?; done)
  echo "Output of booting VM $1 : $start_result"
  echo "Sleeping additional 15 secs after peacefull boot"
  sleep 15
}

function add_keys () {
  for i in `find /var/lib/jenkins/.ssh/id_rsa* | grep -v '.pub'` ; do ssh-add $i ; done
}

#vars

knifeclient_name=lg-head-projectXtest.liquid.glx
headnode_name=projectX-pull-requests

# TEST SCENARIO

cd test/vagrant

# teardown of previous sessions
vagrant destroy projectX-pull-requests-kiosk -f
vagrant destroy projectX-pull-requests-display -f
vagrant destroy $headnode_name -f

echo "Not managing knife client because => chef_zero "
echo "All ssh keys presented below"
ssh-add -l

# headnode
vagrant up ${headnode_name}

# displaynodes

result=$(vboxmanage convertfromraw ../files/ipxe.usb ../files/ipxe_projectX-pull-requests-kiosk.vmdk --format=VMDK ; vagrant up projectX-pull-requests-kiosk; echo $?)
echo "projectX-pull-requests-kiosk : $result"

result=$(vboxmanage convertfromraw ../files/ipxe.usb ../files/ipxe_projectX-pull-requests-display.vmdk --format=VMDK ; vagrant up projectX-pull-requests-display; echo $?)
echo "projectX-pull-requests-display : $result"

# test phase
OPTIONS=`vagrant ssh-config  ${headnode_name} | grep -v ${headnode_name} | awk -v ORS=' ' '{print "-o " $1 "=" $2}'`
scp ${OPTIONS} ../integration/projectX-pr/bats/*.bats vagrant@${headnode_name}:/tmp/bats_tests

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/pre_build_checks.bats'

halt_vm projectX-pull-requests-kiosk
halt_vm projectX-pull-requests-display

echo "Building teh ISO (it may take a long time)"
ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/build_iso.bats'

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/set_grub_to_make_partitions.bats'

echo "Booting nodes"

boot_vm projectX-pull-requests-kiosk
boot_vm projectX-pull-requests-display

echo "Sleeping 30 secs for the DNS to boot and setting the grub to boot the ISO"
sleep 30

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/set_grub_to_boot_the_iso.bats'
echo "Sleeping for 4 mins for the displaynodes to boot fresh ISO"
sleep 240

echo "Running the tests inside the headnode:"

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/post_checks.bats'

So finally we get the following pipeline:

Clone Chef pull request from GitHub
Create Vagrantfile on the basis of Vagrantfile template
Create run_tests.sh script for running the tests
Destroy all previously created Vagrant boxes
Create one Chef Vagrant box
Create ISO Vagrant boxes with ipxe bootloader
Converge the Vagrant box with Chef
Copy BATS tests onto the headnode
Run initial BATS tests that build an ISO
Boot display nodes with the newly created ISO
Run final integration tests on the stack

Elapsed time—between 40 and 50 minutes.

Looking at development environments with DevCamps and Vagrant

2014-08-25T00:00:00+00:00

For most web developers, you have practices and tools that you are used to using to do your work. And for most web developers this means setting up your workstation with all the things you need to do your editing, compiling, testing, and pushing code to some place for sharing or deployment. This is a very common practice even though it is fraught with problems- like getting a database setup properly, configuring a web server, any other services (memcached, redis, mongodb, etc), and many more issues.

Hopefully at some point you realize the pain that is involved in doing everything on your workstation directly and start looking for a better way to do web development. In this post I will be looking at some ways to do this better: using a virtual machine (VM), Vagrant, and DevCamps.

Using a VM for development

One way to improve things is to use a local virtual machine for your development (for example, using VirtualBox, or VMware Fusion). You can edit your code normally on your workstation, but then execute and test it in the VM. This also makes your workstation “clean”, moving all those dependencies (like a database, web server, etc.) off your workstation and into the VM. It also gets your dev environment closer to production, if not identical. Sounds nice, but let’s break down the pros and cons.

Benefits of using a VM

Dev environment closely matches production.
Execute and test code in a dedicated machine (not your workstation directly).
Allows for multiple projects to be worked on concurrently (one VM per project).
Exposes the developer to the Operations (systems administration) side of the web application (always a good thing).
Developer can edit files using their favorite text editor locally on the workstation (but will need to copy files to the VM as needed).

Problems with using a VM

Need to create and configure the VM. This could be very time consuming and error prone.
Still need to install and configure all services and packages. This could also be time consuming and error prone.
Backups of your work/configuration/everything are your own responsibility (extremely unlikely to happen).
Access to your dev environment is extremely limited, thus probably only you can access it and test things on it. No way for a QA engineer or business owner to test/demo your work.
Inexperienced developers can break things, or change them to no longer match production (install arbitrary packages, different versions than what is in production, screw up the db, screw up Apache configuration, etc.).
If working with an established database, then downloading a dump, installing, and getting the database usable is time consuming and error prone. (“I just broke my dev database!” can be a complete blocker for development.)
The developer needs to set up networking for the VM in order to ssh to it, copy files back and forth, and point a web browser to it. This may include manually setting up DNS, or /etc/hosts entries, or port forwarding, or more complex setups.
If using SSL with the web application, then the developer also needs to generate and install the SSL cert and configure the web server correctly.

Vagrant

What is Vagrant? It is a set of tools to make it easier to use a virtual machine for your web development. It attempts to lessen many of the problems listed above through the use of automation. By design it also makes some assumptions about how you are using the VM. For example, it assumes that you have the source code for you project in a directory somewhere directly on your workstation and would prefer to use your favorite text editor on those files. Instead of expecting you to continually push updated files to your VM, it sets up a corresponding directory on the VM and keeps the two in sync for you (using either shared folders, NFS, Samba, or rsync). It also sets up the networking for accessing the VM, usually with port forwarding, so you don’t have to worry about that.

Benefits of Vagrant

Same as those listed above for using a VM, plus…
Flexible configuration (Vagrantfile) for creating and configuring the VM.
Automated networking for the VM with port forwarding. Abstracted ssh access (don’t need to set up a hostname for the VM, simply type vagrant ssh to connect). Port forwarded browser access to the VM (usually http://localhost:8080, but configurable).
Synced directory between your workstation and the VM for source code. Allows for developers to use their favorite text editor locally on their workstation without needing to manually copy files to the VM.
Expects the use of a configuration management system (like puppet, chef, salt, or bash scripts) to “provision” the VM (which could help with proper and consistent setup).
Through the use of Vagrant Cloud you can get a generated url for others to access your VM (makes it publicly available through a tunnel created with the command vagrant share).
Configuration (Vagrantfile and puppet/chef/salt/etc.) files can be maintained/reviewed by Operations engineers for consistency with production.

Problems with Vagrant

Still need to install and configure all services and packages. This is lessened with the use of a configuration management tool like puppet, but you still need to create/debug/maintain the puppet configuration and setup.
Backups of your work/configuration/everything are your own responsibility (extremely unlikely to happen). This may be lessened for VM configuration files, assuming they are included in your project’s VCS repo along with your source code.
Inexperienced developers can still break things, or change them to no longer match production (install arbitrary packages, different versions than what is in production, screw up the db, screw up Apache configuration, etc.).
If working with an established database, then downloading a dump, installing, and getting the database usable is time consuming and error prone. (“I just broke my dev database!” can be a complete blocker for development.)
If using SSL with the web application, then the developer also needs to generate and install the SSL cert and configure the web server correctly. This might be lessened if puppet (or whatever) is configured to manage this for you (but then you need to configure puppet to do that).

DevCamps

The DevCamps system takes a different approach. Instead of using VMs for development, it utilizes a shared server for all development. Each developer has their own account on the camps server and can create/update/delete “camps” (which are self-contained environments with all the parts needed). There is an initial setup for using camps which needs thorough understanding of the web application and all of its dependencies (OS, packages, services, etc.). For each camp, the system will create a directory for the user with everything related to that camp in it, including the web application source code, their own web server configuration, their own database with its own configuration, and any other resources. Each camp is assigned a camp number, and all services for that camp run on different ports (based on the camp number). For example, camp 12 may have Apache running on ports 9012 (HTTP) and 9112 (HTTPS) and MySQL running on port 8912. The developer doesn’t need to know these ports, as tools allow for easier access to the needed services (commands like mkcamp, re for restarting services, mysql_camp for access to the database, etc.).

DevCamps has been designed to address some of the pain usually associated with development environments. Developers usually do not need to install anything, since all dependencies should already be installed on the camps server (which should be maintained by an Operations engineer who can keep the packages, versions, etc. consistent with production). Having all development on a server allows Operations engineers to backup all dev work fairly easily. Databases do not need to be downloaded, manually setup, or anything- they should be set up initially with the camps system and then running mkcamp clones the database and sets it up for you. Running refresh-camp --db allows a developer to delete their camp’s database and get a fresh clone, ready to use.

Benefits of DevCamps

Each developer can create/delete camps as needed, allowing for multiple camps at once and multiple projects at once.
Operations engineers can manage/maintain all dependencies for development, ensuring everything is consistent with production.
Backups of all dev work is easy (Operations engineer just needs to backup the camps server).
Developer does not need to configure services (camp templating system auto-generates needed configuration for proper port numbers), such as Apache, nginx, unicorn, MySQL, Postgres, etc.
SSL certificates can be easily shared/generated/installed/etc. automatically with the mkcamp script. Dev environments can easily have HTTPS without the developer doing anything.
Developers should not have permissions to install/change system packages or services. Thus inexperienced developers should not be able to break the server, other developer’s environments, install arbitrary software. Screwing up their database or web server config can be fixed by either creating a new camp, refreshing their existing one, or an Operations engineer can easily fix it for them (since it is on a central server they would already have access to, and not need to worry about how to access some VM who knows where).

Problems with DevCamps

Since all camps live on a shared server running on different ports, this will not closely match production in that way. However, this may not be significant if nearly everything else does closely match production.
Adding a new dependency (for example, adding mongodb, or upgrading the version of Apache) may require quite a bit of effort and will affect all camps on the server- Operations engineer will need to install the needed packages and add/change the needed configuration to the camps system and templates.
Using your favorite text editor locally on your workstation doesn’t really work since all code lives on the server. It is possible to SFTP files back and forth, but this can be tedious and error prone.
Many aspects of the Operations (systems administration) side of the web application are hidden from the developer (this might also be considered a benefit).
All development is on a single server, which may be a single point of failure (if the camps server is down, then all development is blocked for all developers).
One camp can use up more CPU/RAM/disk/etc. then others and affect the server’s load, affecting the performance of all other camps.

Concluding Thoughts

It seems that Vagrant and DevCamps certainly have some good things going for them. I think it might be worth some thought and effort to try to meld the two together somehow, to take the benefits of both and reduce the problems as much as possible. Such a system might look like this:

Utilize vagrant commands and configuration, but have all VMs live on a central VM server. Thus allowing for central backups and access.
Source code and configuration lives on the server/VM but a synced directory is set up (sshfs mount point?) to allow for local editing of text files on the workstation.
VMs created should have restricted access, preventing developers from installing arbitrary packages, versions, screwing up the db, etc.
Configuration for services (database, web server, etc.) should be generated/managed by Operations engineers for consistency (utilizing puppet/chef/salt/etc.).
Databases should be cloned from a local copy on the VM server, thus avoiding the need to download anything and reducing setup effort.
SSL certs should be copied/generated locally on the VM server and installed as appropriate.
Sharing access to a VM should not depend on Vagrant Cloud, but instead should use some sort of internal service on the VM server to automate VM hostname/DNS for browser and ssh access to the VM.

I’m sure there are more pros and cons that I’ve missed. Add your thoughts to the comments below. Thanks.