Reliable Distributed Job Queue Engine

A production-inspired distributed job queue built with Node.js, PostgreSQL, Redis, React, WebSockets, and Docker. The system ensures reliable background job execution through lease-based processing, automatic recovery, retries with exponential backoff, dead-letter queues, worker heartbeats, and real-time monitoring.

Project Overview

Reliable Distributed Job Queue Engine is a fault-tolerant background job processing system inspired by production queue architectures.

Jobs are stored durably in PostgreSQL, claimed using a lease mechanism, processed by distributed workers, and automatically recovered if a worker crashes. Failed jobs are retried using exponential backoff before being moved to a Dead Letter Queue (DLQ). The system also includes worker heartbeats, Redis Pub/Sub, WebSocket-based live updates, Docker support, and a React monitoring dashboard.

Features

• Durable PostgreSQL-backed job queue

• Lease-based distributed job processing

• Automatic lease recovery

• Retry mechanism with exponential backoff

• Dead Letter Queue (DLQ)

• Replay failed jobs

• Worker registration & heartbeat monitoring

• Automatic offline worker detection

• Redis Pub/Sub event broadcasting

• WebSocket-powered live dashboard

• Health check endpoint

• Dockerized multi-service setup

  System Architecture

flowchart LR
Client["Client / API Request"]
Dashboard["React Dashboard"]
Backend["Node.js Backend"]
Redis["Redis Pub/Sub"]
Postgres["PostgreSQL"]
Worker["Worker Service"]
Recovery["Recovery Worker"]
Client --> Backend
Backend --> Postgres
Backend --> Redis
Worker --> Postgres
Worker --> Redis
Recovery --> Postgres
Redis --> Dashboard

Loading

🔄 Job Lifecycle

Client
│
▼
Create Job
│
▼
PostgreSQL Queue (PENDING)
│
▼
Worker Claims Job (Lease)
│
▼
PROCESSING
│
├──────────────► Success
│ │
│ ▼
│ COMPLETED
│
└──────────────► Failure
│
▼
Retry (Backoff)
│
┌───────────┴───────────┐
│ │
Retry Left Max Retries
│ │
▼ ▼
PENDING Dead Letter Queue

Tech Stack

Category	Technology
Backend	Node.js, Express.js
Frontend	React, Tailwind CSS
Database	PostgreSQL + Prisma ORM
Cache / Messaging	Redis Pub/Sub
Real-time	WebSockets
Containerization	Docker & Docker Compose

Project Structure

Reliable-Distributed-Job-Queue-Engine
│
├── backend
│ ├── prisma
│ ├── src
│ │ ├── controllers
│ │ ├── services
│ │ ├── repositories
│ │ ├── workers
│ │ ├── routes
│ │ └── ...
│ └── Dockerfile
│
├── frontend
│ ├── src
│ └── Dockerfile
│
└── docker-compose.yml

##Getting Started

Prerequisites

• Node.js 22+
• PostgreSQL
• Redis
• Docker (Optional)

Clone the Repository

git clone https://github.com/avent18/Reliable-Distributed-Job-Queue-Engine.git
cd Reliable-Distributed-Job-Queue-Engine

Backend Setup

cd backend
npm install

Create a .env file inside the backend directory and configure your environment variables.

Run database migrations:

npx prisma migrate deploy

Start the backend server:

npm run dev

Frontend Setup

cd frontend
npm install
npm run dev

🐳 Running with Docker

Start the complete application using Docker Compose:

docker compose up --build

This starts:

• Backend API
• React Dashboard
• PostgreSQL
• Redis

Stop all services:

docker compose down

API Overview

Method	Endpoint	Description
POST	/jobs/createjob	Create a new job
GET	/jobs/getall	Retrieve all jobs
GET	/queue/stats	Get queue statistics
GET	/worker/getworkers	List registered workers
POST	/dead-letter/replay/:jobId	Replay a failed job
GET	/health	Health check

✅ Features at a Glance

• Durable PostgreSQL Queue
• Distributed Worker Processing
• Lease-based Job Execution
• Automatic Job Recovery
• Retry with Exponential Backoff
• Dead Letter Queue (DLQ)
• Replay Failed Jobs
• Worker Heartbeats
• Offline Worker Detection
• Redis Pub/Sub
• WebSocket Dashboard
• Dockerized Deployment

Author

Naveen Kumar

If you found this project useful, consider giving it a ⭐ on GitHub.