Configuring Test Kitchen output for Jenkins

In a previous blog post we talked about how we setup Kitchen tests in a Jenkins environment, leaving out a piece of the puzzle that we hadn’t figured out yet: how to have tests results output in a way that can be parsed by Jenkins, with all the advantages that this implies (statistics, test history etc.). In this blog post we discuss a solution we recently found, which although is not final, is our best attempt given the current state of development of the tools involved.

The busser runner we are using for out Kitchen tests is busser-serverspec and at the time of this writing, the problem of having this busser output test results in a format other than the default one, is still open and it’s being tracked on this github issue 

Some of the ideas in this ticket served as a starting point for our approach although we took a slightly different route, specifically in the step that takes care of transferring the junit-formatted output from the machine under test to the Jenkins build agent. 

Here is a way you can get your serverspec Test Kitchen tests output to be successfully parsed by Jenkins.

Configure serverspec to install required dependencies

In order to install a JUnit formatter gem in the machine under test, create a Gemfile inside the serverspec folder of your recipe’s test folder, if you don’t have one already. The Gemfile should contain the following:

source 'https://rubygems.org'

gem ‘yarjuf’

# other dependencies

Yarjuf is a JUnit formatter for RSpec and it produces test results in a format that is fully compatible with Jenkins. If serverspec finds this Gemfile in the correct folder, it will install all of the dependencies defined in there and make them available during the tests.

Configure RSpec to use the JUnit formatter

Once Yarjuf is installed, we can instruct RSpec to use it by including the following lines in the spec_helper.rb file for your test machine:

require ‘yarjuf’

RSpec.configure do |c|

    c.formatter = ‘JUnit’

end

This has the effect that the console output of ‘kitchen test vm-name’ will contain the test results formatted in JUnit. The next step takes care of extracting such report and save it in a xml file that can be directly passed to Jenkins for the parsing step. 

Extract xml report out of Test Kitchen output

This step is where our approach differs from the one advised in the github ticket mentioned above. Instead of having the JUnit formatter write its output on a file local to the machine under test and then transfer the file to the Jenkins test agent, we let the formatter output the report within the console output.

Using the tee command, we copy the console output to a file, and then feed this file to a awk script which takes care of isolating the report into a valid xml file:

kitchen test -d always vm_name | tee kitchen_tests_vm_name.log awk -f filter_junit.awk kitchen_tests_vm_name.log > kitchen_tests_vm_name.xml 

finally, this is the content of the filter_junit.awk script we wrote to extract the JUnit report:

#!/usr/bin/awk -f

BEGIN {

    should_print=0;

} 

{

    if ($0 ~ /\?xml/ ) should_print=1;

    gsub(/^[ \t]+/, “”, $0) 

    if ($0 ~ /<\/testsuites>/ ) { 

        print $0;

        should_print=0; 

    }

    if (should_print) print $0;

} 

Configure Jenkins to parse the serverspec

JUnit xml files As the last step, we need to tell Jenkins where our JUnit reports are located. To do so, in the build configuration page click ‘Add post-build action’ at the bottom, select ‘Publish JUnit test result report’ and edit the location of the files according to your setup.

Conclusions

The steps outlined here helped us getting a step further in our Continuous Integration for our infrastructure code. Certainly having Jenkins understand the test results is much better than having to go through the whole Test Kitchen output to find the results of our tests. Of course this approach is far from ideal, but given the current state of the libraries involved, it was a good compromise for a decent test reporting without excessive effort in the setup. Hopefully this can all be avoided once serverspec will support JUnit reporting and Test Kitchen will accordingly export such reports to testing machine. Until then, please feel free to share with us your ideas and other possible approaches.