fhwang.net

The nuts-and-bolts of configuring Resque and resque-web for a production Rails app

At Profitably, we’ve been using Resque for a few months now, and we’re loving it. Resque’s web interface is a great way to get a high-level understanding of your background workers, and its pluggable design has made it easy for others to contribute a number of really useful plugins. For Profitably, getting control of our background jobs is critical. Our software doesn’t have a ton of public-facing controllers & actions yet, but under the water line it’s already a large system built around mapping and aggregating financial data. (If you’re a Rubyist, and this sounds interesting to you, you should check out our jobs page.)

From a code perspective, it’s not that hard to move to Resque—you write Job classes and you just run them. But I found that I had a few speedbumps in setting it up for production, not least because configuration is not my strong suit. So I decided to describe how we configured Resque to run in production. There are a lot of duct-tapey parts to our setup, and it’s quite likely you’ll make some different choices, but what I describe below has worked well for us so far.

Build a resque-web directory in your Rails app

A lot of the work from setting up Resque is from setting up resque-web, the Sinatra app that helps you monitor Resque.

If you suck at sysadmin like I do, you will be tempted to just run Resque without resque-web. Don’t do that. The easy visibility you get from resque-web can be a lifesaver.

I committed a self-contained resque-web directory in the main Rails app, and then pointed Passenger to it. The resque-web directory looks like this:

$ find ./config/resque-web
config/resque-web
config/resque-web/config.ru
config/resque-web/public
config/resque-web/public/placeholder.txt
config/resque-web/tmp
config/resque-web/tmp/placeholder.txt

It’s weird to put a 2nd web app in the config directory of your Rails app, but it seemed less weird than any other place. If you wanted to be super-tidy about it, you could maybe make the case for a separate project with its own deploy file, but it’s only a few files. Also, keeping it with the Rails app makes it easier to keep it in sync with the Redis config, which I ended up sharing across the Rails app and the resque-web app. More on that below.

For config.ru, copy over the basic config.ru that comes packaged with Resque. I made a few modifications: These are described below. The other two files, config/resque-web/public/placeholder.txt and config/resque-web/tmp/placeholder.txt, are just empty files that will force Git to keep their directories, which are both used by Passenger.

Modify config.ru

You’ll need to modify config.ru from what’s packaged with Resque. In particular, there’s a decent amount of work involved in giving resque-web access to configuration information in the Rails app, since it’s not a Rails app itself. All of the modifications described below should happen before the last two lines, which you should leave alone:

use Rack::ShowExceptions
run Resque::Server.new

Require Resque in config.ru

config.ru has this line for adding Resque to your load path:
$LOAD_PATH.unshift ::File.expand_path(::File.dirname(__FILE__) + '/lib')

You should replace it with lines that find the lib directory wherever you’ve got the Resque gem. In Profitably’s case, we’re not on Bundler yet (I know, embarrassing), and we’ve got our gems in vendor/gems. So our load path config looks like:

gem_path = ::File.expand_path(::File.dirname(__FILE__) + '/../../vendor/gems')
$: << "#{gem_path}/resque-1.10.0/lib"

Require the Redis config in config.ru

Your Redis config is needed in three places:

  1. Rails application servers
  2. Resque background workers
  3. resque-web application

The first and second are easy because they’ll both get the Rails environment. But the third is a little trickier because it won’t be loading your Rails app. So we need to put the Redis config somewhere that’s not tightly bound to Rails, and include it from both your Rails environment and from the resque-web application.

So the steps I took are:

  1. Put the Redis config in config/initializers/redis.rb, where Rails will load it automatically
  2. Require that redis.rb file in config.ru, so resque-web will get it too.
config/initializers/redis.rb looks like this:
rails_env = ENV['RAILS_ENV'] || 'development'

if rails_env == 'production'
  $redis = Redis.new(:host => 'redis1.myapp.com')
elsif rails_env == 'staging'
  $redis = Redis.new(:host => 'redis1-staging.myapp.com')
else 
  $redis = Redis.new
end

Resque.redis = $redis

And I added these lines to config.ru:

# Figure out the Rails environment, defaulting to production
ENV['RAILS_ENV'] ||= 'production'

# Require the redis config
require ::File.expand_path(::File.dirname(__FILE__) + "/../initializers/redis")

It’s a bit messy, because you’ve got a non-Rails-app loading ENV['RAILS_ENV'], but c’est la vie.

Add an HTTP Basic authentication password to config.ru (optional)

These lines add a quick-and-dirty password in case somebody finds out what subdomain we’re using for resque-web:

AUTH_PASSWORD = "top-secret-password" 
if AUTH_PASSWORD
  Resque::Server.use Rack::Auth::Basic do |username, password|
    password == AUTH_PASSWORD
  end
end

Set up a Passenger vhost to start resque-web

Wherever Apache looks for virtual host config files, put a config file for resque-web—for us that’s /etc/apache2/sites-enabled/resque-web, which looks like this:

<VirtualHost *:80>
  ServerName resque-web.myapp.com
  DocumentRoot /u/apps/myapp/current/config/resque-web/public
</VirtualHost>

Passenger knows to look for config.ru in /u/apps/myapp/current/config/resque-web, which it will use to start resque-web.

Running resque-web in environments other than production (optional)

You may want to run resque-web in other deployed environments, such as staging. If you want to pass a different environment to resque-web, you can do so by hard-coding it in your vhost file:

<VirtualHost *:80>
  ServerName resque-web-staging.myapp.com
  RackEnv staging
  DocumentRoot /u/apps/myapp/current/config/resque-web/public
</VirtualHost>

This is visible in config.ru as ENV['RACK_ENV'], so you can copy that into the Rails environment setting by adding to the lines that I described above:

ENV['RAILS_ENV'] ||= ENV['RACK_ENV']
ENV['RAILS_ENV'] ||= 'production'

... and then that gets used by config/initializers/redis.rb.

Setup up your Cap file to restart resque-web when you deploy

Not that resque-web’s going to change that often, but you might as well:

namespace :deploy do
  task :restart_resque_web, :roles => :app, :except => { :no_release => true } do
    run "#{try_sudo} touch #{File.join(
      current_path,'config','resque-web','tmp','restart.txt'
    )}" 
  end
  after 'deploy:restart', 'deploy:restart_resque_web'
end

Loading the Rails environment into your Resque workers

Now that you’re done configuring resque-web, it’s time to set up Resque workers. When you’re starting out you can test them out by running them on the server directly. However, if you just run

RAILS_ENV=production QUEUE=* rake resque:work

Rake and Resque will load, try to run your jobs without your Rails environment, and then spit out a ton of errors. Load the environment into the rake task like so:

RAILS_ENV=production QUEUE=* rake environment resque:work

Setup your workers with multiple queues as pseudo-priorities (optional)

Resque doesn’t support numerical priorities like Delayed::Job, but since it supports multiple queues you can get the same effect with this:

RAILS_ENV=production QUEUE=queue1,queue2,queue3,queue4,queue5 rake resque:work

The workers will take jobs off the queues in order, starting with queue1. In practice, we put most of our jobs into the middle queue, with a few exceptions. It’s also nice to have the option of re-prioritizing everything in production by hand if we have to, by moving it up or down one queue.

Setup a Cap task to restart resque workers on deploy

This step isn’t much different from setting up worker processes for anything else—if you did this for Delayed::Job or another background processing library, this’ll be pretty familiar to you.

A lot of people will do this with something like God, but we did it with PID files. Remembering to load the Rails environment, and using multiple queues, our Cap task looks like this:

namespace :deploy do
  task :restart_resque_workers, :roles => :resque_worker, :except => { :no_release => true } do
    pid_file = "#{current_path}/tmp/pids/resque_worker.pid" 
    run "test -f #{pid_file} && cd #{current_path} && kill -s QUIT `cat #{pid_file}` || rm -f #{pid_file}" 
    queues = (1..5).to_a.map { |i| "queue#{i}" }.join(',')
    run "cd #{current_path}; RAILS_ENV=#{rails_env} QUEUE=#{queues} VERBOSE=1 nohup rake environment resque:work& > #{current_path}/log/resque_worker.log && echo $! > #{pid_file}" 
  end
  after 'deploy:restart', 'deploy:restart_resque_workers'
end

Go to lunch, or have a drink (depending on the time of day)

So that’s about it. It was a decent amount of work getting it set up, but now we have a simple queueing system that doesn’t hit our SQL database, a great top-level interface into the whole system, and we can make use of all those great Resque plugins.

I hope this write-up is helpful, and as always, suggestions are very welcome.

blog comments powered by Disqus

« Previous post

Next post »