Running Resque with God, Redis, and Capistrano - Part Two - God - Blog

This is a second article in the series on how to get Resque and Redis running with God and Capistrano.
See part one

God is a process monitoring framework written in Ruby. Read more about god. The gist of it is - God can start, stop, restart and monitor your applications.

Resque comes with a God script to manage it, so all I needed was to install God as a service and add the Resque script. I followed the instructions from gist with a few modifications of my own.

First, I installed God and made a directory for God's configuration files.

# as root 
$ gem install god
$ cd /opt
$ mkdir god
$ cd god && touch master.conf && touch god-initd.conf 
$ chmod +x god-initd.conf

Then I needed a script to run God as a service. Here is the script, copy it to your god-initd.conf

#!/bin/bash
#
# god       Startup script for god (http://god.rubyforge.org)
#
# chkconfig: - 85 15
# description: God is an easy to configure, easy to extend monitoring \
#              framework written in Ruby.
#

CONF_DIR=/opt/god
GOD_BIN=/usr/local/bin/god
RUBY_BIN=/usr/local/bin/ruby
RETVAL=0

# Go no further if config directory is missing.
[ -d "$CONF_DIR" ] || exit 0

case "$1" in
    start)
      # Create pid directory
      $RUBY_BIN $GOD_BIN -c $CONF_DIR/master.conf
      RETVAL=$?
  ;;
    stop)
      $RUBY_BIN $GOD_BIN terminate
      RETVAL=$?
  ;;
    restart)
      $RUBY_BIN $GOD_BIN terminate
      $RUBY_BIN $GOD_BIN -c $CONF_DIR/master.conf
      RETVAL=$?
  ;;
    status)
      $RUBY_BIN $GOD_BIN status
      RETVAL=$?
  ;;
    *)
      echo "Usage: god {start|stop|restart|status}"
      exit 1
  ;;
esac

exit $RETVAL

The master.conf file should have the load command with the path to your .god configuration file for your application. You can add as many commands to that file as you want.

# /opt/god/master.conf

load "/path/to/your/app/current/config/resque.god"

load "/path/to/another/app/current/config/resque.god"

load "/path/to/cool/app/current/config/mailer.god"

Now, that you have these, lets make God script run as a daemon and start god

$ cp god-initd.sh /etc/init.d/god-monitor
$ chkconfig --add god-monitor
$ chkconfig god-monitor on
$ service god-monitor start

You will need a resque.god script in your configuration directory. You can either use the one provided by Resque in their examples directory or the one below.

God.watch do |w|
  w.name          = 'resque'
  w.interval      = 30.seconds
  w.env           = { 'RAILS_ENV' => 'production',
                      'QUEUE'     => '*' }
  w.uid           = 'user'
  w.gid           = 'user'
  w.dir           = File.expand_path(File.join(File.dirname(__FILE__),'..'))
  w.start         = "/usr/local/bin/rake resque:work"
  w.start_grace   = 10.seconds
  w.log           = File.expand_path(File.join(File.dirname(__FILE__), '..','log','resque-worker.log'))

  # restart if memory gets too high
  w.transition(:up, :restart) do |on|
    on.condition(:memory_usage) do |c|
      c.above = 200.megabytes
      c.times = 2
    end
  end

  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
      c.interval = 5.seconds
    end

    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
      c.interval = 5.seconds
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_running) do |c|
      c.running = false
    end
  end
end

Once you have the script saved in your applications config directory, lets make you Capistrano restart Resque on deploy.

Just copy and paste this Capistrano definition in to your deploy.rb file

# config/deploy.rb

desc "Hot-reload God configuration for the Resque worker"
deploy.task :reload_god_config do
  sudo "god stop resque"
  sudo "god load #{File.join deploy_to, 'current', 'config', 'resque.god'}"
  sudo "god start resque"
end


# Reload the config file for the resque worker after deploy
after :deploy, 'deploy:reload_god_config'

As I've mentioned before, I am not the author of these scripts. I've modified them to suit my environment and hopefully simplified some of the steps. Good luck!

The Third Directive Blog

Running Resque with God, Redis, and Capistrano - Part Two - God 25 Aug Deployment