Graphing Chef Application Deploys with Librato

This is a followup to my previous post on graphing Chef application deploys to Graphite. In my new role we use Librato for time series metrics instead of Graphite. One of the best features in Librato is the concept of metric annotation.  Annotations are very similar to the simple vertical lines I previously used in Graphite, but not only do they display in a more intuitive fashion, they include additional information.  In additional to the vertical line, annotations allow for a name, description, and even a URL to be displayed with the annotation.  This is really useful days / weeks later when you don’t necessarily know what version a line corresponds to.

We’re using annotation within Chef to create annotations for each of our applications during a deploy.  In order to accomplish this I created a LWRP in Chef for Librato metric annotations, and we notify to that LWRP when a deploy action is completed.  You can find the code from the provider below as it’s a bit too large to paste inline with this post.  With our core cookbook included, this provider is now available for any of our application cookbooks to graph deploys.  It can either be notified when a deploy resource updates or it can be used via the “after_deploy proc” functionality in the artifact cookbook, which we use for our application deploys.

artifact_deploy 'my_app' do
  version node[:my_app][:version]
  artifact "http://therealtimsmith.com/artifacts/my_app/#{node[:my_app][:version]}.tar.gz"
  deploy_to '/mnt/my_app'
  owner 'my_app_user'
  group 'my_app_group'

  after_deploy proc {
    if deploy? || manifest_differences?

      # graph the deploy to librato
      mycorecookbook_graph_deploy 'my_app' do
        version node[:my_app][:version]
      end
    end
  }

Deduping

One of the side benefits of all this rich deploy information within the annotations is the ability to de-dupe the annotations.  With multiple servers deploying our application we don’t want to fill our graphs with overlapping, duplicate annotations.  Using the Librato gem we can first query Librato first to see if the annotation exists, and skip adding another annotation if we find anything.  This adds to the complexity of this LWRP over my previous Graphite LWRP, but it makes it much more useful on graphs.

End Result

Librato Deploy

providers/graph_deploy.rb:

# Support whyrun
def whyrun_supported?
  true
end

action :graph do
  if node.chef_environment  == 'production' || node.chef_environment  == 'staging'
    converge_by("Graph deploy of #{@new_resource.name}") do
      graph_deploy
    end
  else
    Chef::Log.info "Not graphing #{@new_resource.name} as we're not in Staging/Prod."
  end
end

def unique_annotation?
  annotations = Librato::Metrics::Annotator.new
  annotation_name = (node.chef_environment + '_' + @new_resource.name + '_deploys').to_sym

  # fetch annotations for this app. rescue because 404 == unique (first ever) annotation
  begin
    host_annotations = annotations.fetch annotation_name, :start_time => (Time.now.to_i - 1800)
  rescue
    return true
  end

  # unassigned is our source. if its key is not there we got back an empty set and we're unique
  return true unless host_annotations['events'].key?('unassigned')

  # if any of the event titles match the desired title then we're not unique
  message = "Graphing deploy of #{@new_resource.name} version #{@new_resource.version}"
  host_annotations['events']['unassigned'].each do |a|
    return false if a['title'] == message
  end

  true
end

# send the annotation to librato if unique
def graph_deploy
  require 'librato/metrics'
  annotation = (node.chef_environment + '_' + @new_resource.name + '_deploys').to_sym
  creds = Chef::EncryptedDataBagItem.load('librato', 'keys')
  begin
    Librato::Metrics.authenticate creds['username'], creds['token']

    # bail out if this deploy has already been graphed
    return unless  unique_annotation?

    Librato::Metrics.annotate annotation, "Deployed #{@new_resource.name} version #{@new_resource.version}", :source => node.name
    Chef::Log.info "Graphing deploy of #{@new_resource.name} version #{@new_resource.version}"
  rescue
    Chef::Log.error 'An error occurred when trying to graph the deploy with Librato'
  end
end

 resources/graph_deploy.rb

actions :graph

default_action :graph

attribute :name, :name_attribute => true, :kind_of => String, :required => true
attribute :version, :kind_of => String, :required => true

attr_accessor :exists

 

Important Note:

This provider requires the librato-metrics gem to be loaded into your omnibus Chef install.  In order to do this you either need to do a chef gem install during compile time or you need to set the following attribute with the chef-client cookbook:

default[‘chef_client’][‘load_gems’] = %w(librato-metrics)

Leave a Reply

Your email address will not be published. Required fields are marked *