Category: Chef

Graphing Chef Application Deploys with Librato

This is a followup to my previous post on graphing Chef application deploys to Graphite. In my new role we use Librato for time series metrics instead of Graphite. One of the best features in Librato is the concept of metric annotation.  Annotations are very similar to the simple vertical lines I previously used in Graphite, but not only do they display in a more intuitive fashion, they include additional information.  In additional to the vertical line, annotations allow for a name, description, and even a URL to be displayed with the annotation.  This is really useful days / weeks later when you don’t necessarily know what version a line corresponds to.

We’re using annotation within Chef to create annotations for each of our applications during a deploy.  In order to accomplish this I created a LWRP in Chef for Librato metric annotations, and we notify to that LWRP when a deploy action is completed.  You can find the code from the provider below as it’s a bit too large to paste inline with this post.  With our core cookbook included, this provider is now available for any of our application cookbooks to graph deploys.  It can either be notified when a deploy resource updates or it can be used via the “after_deploy proc” functionality in the artifact cookbook, which we use for our application deploys.

artifact_deploy 'my_app' do
  version node[:my_app][:version]
  artifact "http://therealtimsmith.com/artifacts/my_app/#{node[:my_app][:version]}.tar.gz"
  deploy_to '/mnt/my_app'
  owner 'my_app_user'
  group 'my_app_group'

  after_deploy proc {
    if deploy? || manifest_differences?

      # graph the deploy to librato
      mycorecookbook_graph_deploy 'my_app' do
        version node[:my_app][:version]
      end
    end
  }

Deduping

One of the side benefits of all this rich deploy information within the annotations is the ability to de-dupe the annotations.  With multiple servers deploying our application we don’t want to fill our graphs with overlapping, duplicate annotations.  Using the Librato gem we can first query Librato first to see if the annotation exists, and skip adding another annotation if we find anything.  This adds to the complexity of this LWRP over my previous Graphite LWRP, but it makes it much more useful on graphs.

End Result

Librato Deploy

providers/graph_deploy.rb:

# Support whyrun
def whyrun_supported?
  true
end

action :graph do
  if node.chef_environment  == 'production' || node.chef_environment  == 'staging'
    converge_by("Graph deploy of #{@new_resource.name}") do
      graph_deploy
    end
  else
    Chef::Log.info "Not graphing #{@new_resource.name} as we're not in Staging/Prod."
  end
end

def unique_annotation?
  annotations = Librato::Metrics::Annotator.new
  annotation_name = (node.chef_environment + '_' + @new_resource.name + '_deploys').to_sym

  # fetch annotations for this app. rescue because 404 == unique (first ever) annotation
  begin
    host_annotations = annotations.fetch annotation_name, :start_time => (Time.now.to_i - 1800)
  rescue
    return true
  end

  # unassigned is our source. if its key is not there we got back an empty set and we're unique
  return true unless host_annotations['events'].key?('unassigned')

  # if any of the event titles match the desired title then we're not unique
  message = "Graphing deploy of #{@new_resource.name} version #{@new_resource.version}"
  host_annotations['events']['unassigned'].each do |a|
    return false if a['title'] == message
  end

  true
end

# send the annotation to librato if unique
def graph_deploy
  require 'librato/metrics'
  annotation = (node.chef_environment + '_' + @new_resource.name + '_deploys').to_sym
  creds = Chef::EncryptedDataBagItem.load('librato', 'keys')
  begin
    Librato::Metrics.authenticate creds['username'], creds['token']

    # bail out if this deploy has already been graphed
    return unless  unique_annotation?

    Librato::Metrics.annotate annotation, "Deployed #{@new_resource.name} version #{@new_resource.version}", :source => node.name
    Chef::Log.info "Graphing deploy of #{@new_resource.name} version #{@new_resource.version}"
  rescue
    Chef::Log.error 'An error occurred when trying to graph the deploy with Librato'
  end
end

 resources/graph_deploy.rb

actions :graph

default_action :graph

attribute :name, :name_attribute => true, :kind_of => String, :required => true
attribute :version, :kind_of => String, :required => true

attr_accessor :exists

 

Important Note:

This provider requires the librato-metrics gem to be loaded into your omnibus Chef install.  In order to do this you either need to do a chef gem install during compile time or you need to set the following attribute with the chef-client cookbook:

default[‘chef_client’][‘load_gems’] = %w(librato-metrics)

Only Deploying When You Want To With Chef aka Don’t Break Prod

Continuous Chef Runs

By design Chef’s client application, chef-client, runs as a service on systems in a chef managed environment with a chef-client run occurring every 20 minutes.  The continuous chef-client runs ensure that systems are always configured as expected and changes can easily be pushed out with quick convergence.  The downside is that any action in a Chef cookbook will run every 20 minutes including the deployment of product code if you use chef to deploy your actual application code.  This could lead to your web application being reinstalled and restarted every 20 minutes unless you’re careful.

 

Preventing Accidental Continuous Deployment

To avoid continuously deploying product you can wrap any process that would cause impact to customers in a “deploy flag”.  This allows you to use Chef’s 20 minute continuous runs to ensure your system is in the appropriate state, while not deploying code.   When you wish to run through the complete process of building a working system from scratch, including pulling down new application code, you can simply set an environmental variable of deploy_build=true.  This can be done remotely during code rollouts via Capistrano or Rundeck to allow for a orchestrated deploy of code.

 

Using the Deploy Flag

At the beginning of your recipe include code as follows to write out the state of the flag and then execute any uninstall recipes or resources.

log "Deploy build is #{ENV["deploy_build"]}"

if ENV["deploy_build"] == "true" then
  include_recipe "COOKBOOKNAME::uninstall"
end

Further in your code when you would like to actually deploy components. (delete directories, copy in files, or anything else that would be customer impacting)

if ENV["deploy_build"] == "true" then
  DEPLOY ME HERE
end

 

End Result

Now running “chef-client” will ensure your system is in the appropriate state, while running “deploy_build=”true” chef-client will run a full deploy of your application.