ResqueRing provides cluster-friendly, auto-scaling, lightweight management services for Resque workers. It can manage multiple worker groups, each watching multiple queues, and scale workers based on available work.
- Multiple worker groups, each supporting multiple queues
- Scale easily from MIN to MAX workers based on queue sizes
- Scale down to MIN workers when idle (even 0!)
- Configurable spawn throttling
- Contractor Mode: start 1st worker automatically when items enter its queues.
- Automatically manages and keeps track of workers it spawns
- Track jobs processed, age & other data for each worker
- Number of jobs processed
- Age of worker
- Memory usage
- Time to process
- All communication happens through Redis for easy clustering
- Manage worker pool sizes locally (1 server) and globally (across servers)
The simplest way to install ResqueRing is to use Bundler.
Add ResqueRing to a Gemfile in your project’s root:
gem'resque_ring'then install it by running Bundler:
$ bundleResqueRing is run from the command line (or a process monitor like monit). Please open your terminal and go to your project's directory.
Running resque_ring with no arguments prints this:
Commands:
resque_ring help [COMMAND] # Describe available commands or one specific command
resque_ring start -c, --config=CONFIG # start a resque_ring
resque_ring version # print versionYou can always get help on the available tasks with the help task:
$ bundle exec resque_ring helpTo get help for a specific command, append it to the end of the previous example, e.g.,
$ bundle exec resque_ring help startWhen started on multiple servers using the same configuration and Redis server, each ResqueRing will coordinate efforts with the others.
ResqueRing is controlled primarily through a configuration file, so starting a new instance is simple:
bundle exec resque_ring start -c my_config.ymlSets the location of the ResqueRing config file. If this is not
specified, ResqueRing will look for a resque_ring.yml file in your current
directory.
Prints the current version of ResqueRing.
bundle exec resque_ring versionYou can control ResqueRing with POSIX signals.
Quit ResqueRing after cleaning up.
$ kill -INT <resque_ring_pid>TERM and QUIT also work splendidly.
Quit all workers cleanly, but leave everything running. Workers will respawn automatically.
$ kill -USR1 <resque_ring_pid>Quit all workers cleanly, reload config, and start fresh.
$ kill -HUP <resque_ring_pid>Quit all workers and prevent them from respawning.
$ kill -STOP <resque_ring_pid>Allow a paused ResqueRing process to go back to work.
$ kill -CONT <resque_ring_pid>The configuration file has three main groups: global settings, redis settings, and worker definitions.
Global settings are at the root of the yml file. All global settings are optional, but ResqueRing will use its own defaults if not set.
delay: 60# seconds to wait before checking againRedis settings, if included, should include the host & port
of your Redis server. If not included, ResqueRing will use
the default Redis host and port: localhost:6379.
redis:
host: localhostport: 6379The workers group can contain any number of subsections, identified by your desired name for each group of workers.
This is useful if you need to manage different sets of Resque workers for different tasks.
Each group of workers has a number of options:
| Option | Note |
|---|---|
| wait_time | how long to wait before starting another worker. |
| threshold | how many items in queue before starting more than min workers. |
| spawn_rate | when spawning new workers, how many to start at a time. |
| remove_when_idle | start removing workers when queue is empty (default: true) |
| spawner | options describing how to start an actual worker task. |
| queues | a list of queues this worker group should watch. |
| pool | options for the pool of workers. |
command is the specific command used to start the worker.dir tells ResqueRing to cwd to this directory before running the command.env provides a list of environment variables set before running the worker.
Note: ResqueRing resets all existing environment variables prior to setting the environment variables specified here.
spawner:
command: bundle exec rake resque:workdir: /this/is/my/work/direnv:
rails_env: developmentA list of queues this worker group is expected to watch.
queues:
- queue_the_first
- queue_tee_pie
- queue_the_musicPool options configure the management of a pool of workers.
min sets the number of workers to start immediately.max sets the maximum number of workers to spawn on this server.global_max sets the maximum number of workers to run across all servers.first_at (see note)
Note: first_at, when used with
min_workers: 0allows a ResqueRing pool to "idle" with no workers when there are no jobs to process. This is Contractor mode.In normal operation, when min_workers is greater than zero, ResqueRing will start
min_workersnumber of workers initially, but will not start more untilthresholdis reached.In Contractor mode, a worker is started immediately as soon as
first_atnumber of items enter the queue, and no more workers will start untilthresholdis reached.Combined with
remove_when_idle, this allows ResqueRing to spin up and down as needed, rather than maintain an always-active pool of workers.
pool:
min: 1# How many to start initially, 0 means no workers until queue; defaults to 1max: 5# The most we'll ever start; defaults to 5global_max: 15# Max workers across all servers; default 0 (no limit)first_at: 1# Use with min_workers 0; fewer than fire_at jobs in queue will not start any workers. Defaults to 1, and is only checked if min_workers is 0A complete configuration file looks something like this:
delay: 60# seconds to wait before checking againredis:
host: localhostport: 6379workers:
indexing: # What you name, you love.wait_time: 120# don't start another worker more often than this (seconds); will only ever start one worker per configured queue within time set above in delaythreshold: 100# If queue gets bigger than this, start another worker until max workers reachedspawn_rate: 1# How many workers to spawn at a time, defaults to 1remove_when_idle: true # start removing workers when queue is idle (defaults to true)spawner:
command: bundle exec rake resque:workdir: /this/is/my/work/direnv:
rails_env: developmentqueues: # list of queues this worker listens for
- queue_the_first
- queue_tee_pie
- queue_the_musicpool:
min: 1# How many to start initially, 0 means no workers until queue; defaults to 1max: 5# The most we'll ever start; defaults to 5global_max: 15# Max workers across all servers; default 0 (no limit)first_at: 1# Use with min_workers 0; fewer than fire_at jobs in queue will not start any workers. Defaults to 1, and is only checked if min_workers is 0- Ability to kill & respawn workers based on metrics above
- Wildcard queues?
Fixing a bug? Adding an awesome new feature? Your pull requests are welcome!
- Please create a feature branch each separate change
- Write tests for your changes! All specs and code quality checks must pass on Travis CI.
- Update the documentation.
- Update the README.
- Please do not change the version number.
Released under the MIT License. See the LICENSE file for further details.