Category: Chef

Graphing Chef Application Deploys with Librato

This is a followup to my previous post on graphing Chef application deploys to Graphite. In my new role we use Librato for time series metrics instead of Graphite. One of the best features in Librato is the concept of metric annotation.  Annotations are very similar to the simple vertical lines I previously used in Graphite, but not only do they display in a more intuitive fashion, they include additional information.  In additional to the vertical line, annotations allow for a name, description, and even a URL to be displayed with the annotation.  This is really useful days / weeks later when you don’t necessarily know what version a line corresponds to.

We’re using annotation within Chef to create annotations for each of our applications during a deploy.  In order to accomplish this I created a LWRP in Chef for Librato metric annotations, and we notify to that LWRP when a deploy action is completed.  You can find the code from the provider below as it’s a bit too large to paste inline with this post.  With our core cookbook included, this provider is now available for any of our application cookbooks to graph deploys.  It can either be notified when a deploy resource updates or it can be used via the “after_deploy proc” functionality in the artifact cookbook, which we use for our application deploys.

artifact_deploy 'my_app' do
  version node[:my_app][:version]
  artifact "{node[:my_app][:version]}.tar.gz"
  deploy_to '/mnt/my_app'
  owner 'my_app_user'
  group 'my_app_group'

  after_deploy proc {
    if deploy? || manifest_differences?

      # graph the deploy to librato
      mycorecookbook_graph_deploy 'my_app' do
        version node[:my_app][:version]


One of the side benefits of all this rich deploy information within the annotations is the ability to de-dupe the annotations.  With multiple servers deploying our application we don’t want to fill our graphs with overlapping, duplicate annotations.  Using the Librato gem we can first query Librato first to see if the annotation exists, and skip adding another annotation if we find anything.  This adds to the complexity of this LWRP over my previous Graphite LWRP, but it makes it much more useful on graphs.

End Result

Librato Deploy


# Support whyrun
def whyrun_supported?

action :graph do
  if node.chef_environment  == 'production' || node.chef_environment  == 'staging'
    converge_by("Graph deploy of #{}") do
  else "Not graphing #{} as we're not in Staging/Prod."

def unique_annotation?
  annotations =
  annotation_name = (node.chef_environment + '_' + + '_deploys').to_sym

  # fetch annotations for this app. rescue because 404 == unique (first ever) annotation
    host_annotations = annotations.fetch annotation_name, :start_time => ( - 1800)
    return true

  # unassigned is our source. if its key is not there we got back an empty set and we're unique
  return true unless host_annotations['events'].key?('unassigned')

  # if any of the event titles match the desired title then we're not unique
  message = "Graphing deploy of #{} version #{@new_resource.version}"
  host_annotations['events']['unassigned'].each do |a|
    return false if a['title'] == message


# send the annotation to librato if unique
def graph_deploy
  require 'librato/metrics'
  annotation = (node.chef_environment + '_' + + '_deploys').to_sym
  creds = Chef::EncryptedDataBagItem.load('librato', 'keys')
    Librato::Metrics.authenticate creds['username'], creds['token']

    # bail out if this deploy has already been graphed
    return unless  unique_annotation?

    Librato::Metrics.annotate annotation, "Deployed #{} version #{@new_resource.version}", :source => "Graphing deploy of #{} version #{@new_resource.version}"
    Chef::Log.error 'An error occurred when trying to graph the deploy with Librato'


actions :graph

default_action :graph

attribute :name, :name_attribute => true, :kind_of => String, :required => true
attribute :version, :kind_of => String, :required => true

attr_accessor :exists


Important Note:

This provider requires the librato-metrics gem to be loaded into your omnibus Chef install.  In order to do this you either need to do a chef gem install during compile time or you need to set the following attribute with the chef-client cookbook:

default[‘chef_client’][‘load_gems’] = %w(librato-metrics)

Only Deploying When You Want To With Chef aka Don’t Break Prod

Continuous Chef Runs

By design Chef’s client application, chef-client, runs as a service on systems in a chef managed environment with a chef-client run occurring every 20 minutes.  The continuous chef-client runs ensure that systems are always configured as expected and changes can easily be pushed out with quick convergence.  The downside is that any action in a Chef cookbook will run every 20 minutes including the deployment of product code if you use chef to deploy your actual application code.  This could lead to your web application being reinstalled and restarted every 20 minutes unless you’re careful.


Preventing Accidental Continuous Deployment

To avoid continuously deploying product you can wrap any process that would cause impact to customers in a “deploy flag”.  This allows you to use Chef’s 20 minute continuous runs to ensure your system is in the appropriate state, while not deploying code.   When you wish to run through the complete process of building a working system from scratch, including pulling down new application code, you can simply set an environmental variable of deploy_build=true.  This can be done remotely during code rollouts via Capistrano or Rundeck to allow for a orchestrated deploy of code.


Using the Deploy Flag

At the beginning of your recipe include code as follows to write out the state of the flag and then execute any uninstall recipes or resources.

log "Deploy build is #{ENV["deploy_build"]}"

if ENV["deploy_build"] == "true" then
  include_recipe "COOKBOOKNAME::uninstall"

Further in your code when you would like to actually deploy components. (delete directories, copy in files, or anything else that would be customer impacting)

if ENV["deploy_build"] == "true" then


End Result

Now running “chef-client” will ensure your system is in the appropriate state, while running “deploy_build=”true” chef-client will run a full deploy of your application.