“Trust but Verify” Your Chef Infrastructure

A cornerstone of infrastructure as code is treating our infrastructure as we would any other software project by thoroughly testing all changes. As Chef users we have a plethora of options when it comes to testing infrastructure code. We can ensure teams use agreed upon coding standards by linting with Rubocop and Foodcritic.  We can test the outcome of complex cookbook logic quickly with unit testing in Chefspec.  We can perform lengthy, but thorough integration tests with bats, Serverspec, or Inspec running from within Test Kitchen. Combined these tests ensure high quality code powering our infrastructure.

Where testing tends to get complex is how we bring those various frameworks together so that any change to our infrastructure is tested before it enters production. In particular, for those of us that use monolithic repository structure for our chef environments, roles, and cookbooks testing is particularly tricky.  With a monolithic repository all code is stored in a single git repository that a team works out of.  This makes it difficult to determine what chef assets to test as a pull requests comes in. While we may find it acceptable to run full integration tests after changing a single cookbook, we can’t afford the time it would take to run integration tests on every cookbook in our repo when simple change is made. We decompose the changes in a commit to determine which assets have been changed so we can test just those individual assets.

To test just the assets that change in a monolithic repository I wrote Reagan.  Reagan is built to “Trusty but Verify” Chef repository pull requests in Github using Jenkins.  A Jenkins job polls your chef repository for new pull requests to test.  When a new PR is opened Reagan retrieves the list of changed files in the pull request from the Jenkins API.  From that list it builds a list of assets that have been changed.  For data bags, environments, and roles it runs simple JSON validation.  For cookbooks Reagan runs “knife cookbook test” and then checks the chef server to ensure the cookbook version has been incremented.  After performing those basic sanity tests it also runs tests defined in a per cookbook YAML file.  This allows you to vary your testing methods depending on the code.  The reagan_test.yml config contains a simple list of commands to run like this:

  - rubocop .
  - foodcritic . -f any
  - rspec

Jenkins then updates the PR testing status and depending on configuration may send e-mails or post to chat systems like Slack.  Here’s an example of a successful lint test run by Reagan:


Here’s an example of a similar test failing due to a Rubocop offense:


Reagan helped me to ensure quality code entered our production environment without adding burdensome process.  The source is available at https://github.com/tas50/reagan with setup instructions in the Readme.  The gem is available on Rubygems as well.

Leave a Reply

Your email address will not be published. Required fields are marked *