A CLI-based background job queue with retries, exponential backoff, a Dead Letter Queue, and multi-worker support — built in Node.js.
git clone https://github.com/Unseencoderz/QueueCTL
cd QueueCTL
npm install
node bin/QueueCTL.js enqueue '{"id":"job1","command":"echo hello"}'
node bin/QueueCTL.js worker start --count 1
node bin/QueueCTL.js statusRequires Node.js 18+ and npm.
npm installRun directly with Node:
node bin/QueueCTL.js --helpOr link it as a global command:
npm link
QueueCTL --helpRuntime data lives in data/jobs.json, data/config.json, and data/workers.json — created automatically on first run.
| Command | Description |
|---|---|
QueueCTL enqueue '<json>' | Add a new job to the queue |
QueueCTL worker start --count <n> | Start n worker processes |
QueueCTL worker stop | Gracefully stop running workers (finishes current job first) |
QueueCTL status | Show job counts per state + active workers |
QueueCTL list --state <state> | List jobs, optionally filtered by state |
QueueCTL dlq list | List jobs in the Dead Letter Queue |
QueueCTL dlq retry <id> | Reset a dead job back to pending |
QueueCTL config set <key> <value> | Set max-retries or backoff-base |
Run QueueCTL <command> --help for full option details.
Configure retries, then enqueue and run a job:
$ node bin/QueueCTL.js config set max-retries 2
Config max-retries set to 2
$ node bin/QueueCTL.js enqueue '{"id":"readme-ok","command":"echo readme"}'
Job readme-ok enqueued (state: pending)
$ node bin/QueueCTL.js worker start --count 1
readme
1 completed, 0 failed
$ node bin/QueueCTL.js status
pending: 0
processing: 0
completed: 1
failed: 0
dead: 0
active workers: 0A job that keeps failing moves to the DLQ, and can be retried from there:
$ node bin/QueueCTL.js enqueue '{"id":"readme-dead","command":"definitely-missing-QueueCTL-command"}'
Job readme-dead enqueued (state: pending)
$ node bin/QueueCTL.js worker start --count 1
0 completed, 1 failed
$ node bin/QueueCTL.js dlq list
┌─────────┬───────────────┬────────┬───────────────────────────────────────┬──────────┐
│ (index) │ id │ state │ command │ attempts │
├─────────┼───────────────┼────────┼───────────────────────────────────────┼──────────┤
│ 0 │ 'readme-dead' │ 'dead' │ 'definitely-missing-QueueCTL-command' │ 1 │
└─────────┴───────────────┴────────┴───────────────────────────────────────┴──────────┘
$ node bin/QueueCTL.js dlq retry readme-dead
Job readme-dead moved back to pendingListing and stopping workers:
$ node bin/QueueCTL.js list --state pending
┌─────────┬───────────────┬───────────┬──────────┬───────────────────────────────────────┐
│ (index) │ id │ state │ attempts │ command │
├─────────┼───────────────┼───────────┼──────────┼───────────────────────────────────────┤
│ 0 │ 'readme-dead' │ 'pending' │ 0 │ 'definitely-missing-QueueCTL-command' │
└─────────┴───────────────┴───────────┴──────────┴───────────────────────────────────────┘
$ node bin/QueueCTL.js worker stop
No running workers foundNote: error text for a missing command (e.g.
'X' is not recognized...vscommand not found) depends on the host OS shell — see Assumptions below.
enqueue → pending → [worker claims job] → processing
├── exit 0 → completed
├── exit ≠ 0, retries left → pending (delayed via next_run_at)
└── exit ≠ 0, retries exhausted → dead → (dlq retry) → pending
- Jobs are stored as a JSON array in
data/jobs.json; config indata/config.json. - Writes use a temp-file + rename pattern, with a
data/jobs.json.lockfile guarding concurrent read-modify-write updates from multiple workers. worker start --count NforksNchild Node processes. Each claims one job at a time viatryClaimJob(workerId), executes its command, and updates its own state. Worker PIDs are tracked indata/workers.jsonsoworker stopcan signal them.
Full diagrams (component view + state machine) are in design.md.
- JSON file storage, not SQLite, to keep the project small and easy to inspect/explain.
- One child process per worker for true parallel execution, instead of a custom in-process scheduler.
- File-lock based concurrency, not database transactions — sufficient at this
scale, but a crashed worker can leave a job stuck in
processing(locked_byset). Crash recovery for abandoned jobs is not implemented. - Backoff is deterministic:
backoff_base ^ attemptsseconds. Jobs are skipped by the worker until theirnext_run_athas passed. - Commands run via
spawn(cmd, { shell: true }), so command syntax and "not found" error text depend on the host OS/shell.
npm test# or
bash test/validate.shThe script backs up data/jobs.json, data/config.json, and data/workers.json,
runs end-to-end scenarios (success, retry-to-DLQ, parallel workers, invalid command,
persistence across restarts), then restores your original data.
On Windows, run it from an environment with Bash (Git Bash or WSL).
MIT