Configuring Test Kitchen on Jenkins
Test Kitchen proved to be an invaluable tool in our infrastructure workflow, but we have found it can be tricky to integrate into continuous integration systems, so we decided to share our experience setting it up under Jenkins.
Update: you can find a follow-up for this blog post here
We started out by running kitchen tests locally, on our development machines, agreeing to make sure the tests passed every time we made changes to our cookbooks. This quickly became impractical, both because of the growing number of tests and because of the long running times that these tests imply.
Furthermore, we had decided to build our test suite on a very interesting tool called Leibniz, by Stephen Nelson-Smith, the author of “Test-driven infrastructure with Chef”. This is basically a glue layer between cucumber and test kitchen, and it enabled us to develop our infrastructure using the Behavior Driven Development approach that we are growingly familiar with.
By the time we had built a conspicuous test suite, though, this project had become inactive and after a while we decided to migrate our tests to pure Kitchen tests.
This was a chance to review our testing strategy that eventually led to our current setup, a Jenkins build that automatically runs all of our infrastructure tests and is mainly based on the following tools:
- Test Kitchen, automating the creation of test machines for different platforms
- Vagrant, used as a driver for Test Kitchen, is in charge of actually instantiating and managing the machine’s state
- Chef, used to provision the test machine bringing it into the desired state, so it can be tested as necessary
- Libvirt, the virtualization solution that we adopted for the test machines
Here is how to setup a Jenkins build to run Kitchen tests using Vagrant and libvirt.
In our setup we used two separate machines: one is a VM running Jenkins and one is a host machine in charge of hosting the test machines. We recommend a similar setup as it simplifies the virtualization step and also greatly improves performance.
Install Vagrant on the host machine
Although some Linux distributions like Debian or Ubuntu have Vagrant available through their official repository, it is recommended to install Vagrant using the official packages available on the Vagrant website download page.
In order to use libvirt as virtualization solution for the test VMs, a few Vagrant plugins are necessary:
- vagrant-libvirt adds a Libvirt provider to Vagrant
- vagrant-mutate Given the scarce availability of Vagrant boxes for Libvirt, this plugin allows to adapt boxes originally prepared for other providers (e.g. Virtualbox) to Libvirt
This is an optional step, but it is highly recommended as it isolates the ruby installation used by this build from the system ruby and simplifies maintenance as well as troubleshooting.
git clone https://github.com/sstephenson/rbenv.git ~/.rbenv echo ‘export PATH=“$HOME/.rbenv/bin:$PATH”’ >> ~/.bashrc echo ‘eval “$(rbenv init -)”’ >> ~/.bashrc exec $SHELL git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build rbenv install
make sure to replace the ruby version you need in the last command
Install the rbenv Jenkins plugin
This plugin is only needed in case a rbenv has been configured in the previous step. It can be used to instruct the Jenkins build to use a specific rbenv instead of the system’s one. This plugin can be easily installed using Jenkins’ plugin management interface.
Configure the Jenkins build
Create a new Jenkins build following these steps as a guideline:
- New Item
- Freestyle Project
- Make sure the build only runs on the slave running on the host machine using the ‘Restrict where this project can be run’ section
- configure the repository to use for the build and its triggers
- if you installed the rbenv plugin, mark the checkbox ‘rbenv build wrapper’ and select the ruby version you want to use for this build and make sure that the ‘Preinstall gem list’ includes ‘bundler’
- add a build step of type ‘Execute shell’ to install the ruby dependencies:
cd /path/to/cookbook_s_Gemfile; bundle install;
- add another build step of type ‘Execute shell’ to actually execute the tests:
cd /path/to/cookbook; kitchen test -d always
this step assumes that your Gemfile includes the gems necessary to run kitchen tests:
gem ‘test-kitchen’ gem ‘kitchen-vagrant’
the ‘-d always’ option is added to make sure that test machines get destroyed regardless of the test outcome, remove it if you need to inspect the machines for debugging purposes when tests fail and kitchen will leave the machine running and accessible at the end of the test.
Prepare Vagrant boxes
This step takes care of preparing the boxes used during the tests migrating them from other providers to libvirt. Assuming, for example, that your .kitchen.yml file defines a Debian platform like this:
platforms: - name: debian-8.1
you can download a Debian 8.1 box originally prepared for virtualbox with the following command:
vagrant box add ospcode-debian-8.1 box_url
where box_url should be updated to point to a valid box url (the boxes normally used by Test Kitchen can be found here)
once the box has been added to the Vagrant boxes set, it can be adapted for Libvirt like this:
vagrant mutate ospcode-debian-8.1 libvirt
and this base box is now ready for use in tests.
Repeat this step for all of the platforms declared in your .kitchen.yml file.
Configure Kitchen to use Libvirt
By default, Test Kitchen will try to use virtualbox as provider and will bail out if it does not find it. To instruct Test Kitchen that it should use libvirt instead, there are several alternatives, and we picked one that uses the .kitchen.yml file.
Edit your .kitchen.yml file so that the driver section looks like this:
driver: name: vagrant provider: <%= ENV[‘KITCHEN_PROVIDER’] || ‘virtualbox’ %>
This tells Kitchen that if it finds an environment variable called KITCHEN_PROVIDER it will use its value as a provider for the tests, otherwise will fallback to virtualbox. Make sure you add this line to the second step of your Jenkins build before the tests are launched:
The build is now ready to run your kitchen tests.
The actual tests
As mentioned earlier, we started out test suite using Leibniz. This is a very powerful tool that can express from a very high level, in a human readable format, what is required from a certain machine. Eventually, though, we started having several issues with it, and after a few attempts at fixing those issues ourselves by forking the project, we eventually decided to switch to something else. Our choice was first BATs tests and then Serverspec. Serverspec is, as of today, our framework of choice for testing our infrastructure with its expressive and comprehensive set of resource types.
In order to troubleshoot issues that can arise during the convergence of test machines or during the tests, two environment variables can be used that instruct respectively test kitchen and Vagrant to be more verbose about their output:
export KITCHEN_LOG=‘debug’ export VAGRANT_LOG=‘debug’
depending on the situation it might be sensible to enable one or the other, as both can flood your output enough to bury important messages away.
The most obvious and urgent matter we have to fix next is the fact that the outcome of our serverspec tests is currently output in a way that is good for humans to read but not very good for Jenkins to parse (see this issue on GitHub). Hence we will have to find a way to instruct kitchen to, in turn, instruct serverspec to produce JUnit-style test reports that Jenkins can easily parse.
An additional improvement can take advantage of the possibility of creating custom packer boxes that have basic common software and configuration already pre-installed. This can noticeably speed up the time to prepare the test machines during our builds. More information about Packer can be found on the official Packer website.
Furthermore, a possible performance bump can be obtained by caching as much as possible of the resources that each test machine downloads every single time the tests run, like software update packages, gems and so on.
In this regard, the vagrant-cachier plugin for Vagrant looks like the perfect tool for the job.
During the setup of our test configuration, we have found a great deal of help and suggestions at the following links:
If you want to know more about our setup, please don’t hesitate to ask in the comments section below.
- Django-treebeard and Wagtail page creation
- The Charity Sport Tournament in Lublin
- New Release of Agilo for Trac (0.9.15/1.3.15)
- Incontro DevOps Italia 2016
- Configuring Test Kitchen output for Jenkins
- Configuring Test Kitchen on Jenkins
- Better infrastructure management a.k.a. IAC (Infrastructure as Code)
- Our approach to automated visual regression testing
- Test parallelization with Lettuce, take 2
- New Release of Agilo for Trac (0.9.14/1.3.14)