Distributed Job Scheduler

A production-inspired distributed job scheduling platform — users authenticate, create projects, each project owns queues, queues hold jobs (immediate, delayed, scheduled-cron, batch). A pool of independent worker processes atomically claims jobs from queues, executes them concurrently under per-queue concurrency limits, retries failures with configurable backoff, and dead-letters permanent failures. Every step is observable via structured logs, Prometheus metrics, and Grafana dashboards.

Stack: Rust (Axum / Tokio / SQLx) · TypeScript + Vite (React) · PostgreSQL 16 · Docker Compose

Quick Start

Prerequisites

Docker and Docker Compose v2+
OR Rust 1.75+ and Node.js 20+ for local development

Run with Docker Compose (recommended)

# Bring up the entire stack
docker compose up --build
# Scale workers independently
docker compose up --build --scale worker=3

The stack will be available at:

Service	URL	Auth
Frontend dashboard	http://localhost:8080	admin@example.com / password123
API server	http://localhost:3000	JWT Bearer token
Prometheus	http://localhost:9090	—
Grafana	http://localhost:3001	admin / admin

Run locally (development)

# 1. Start PostgreSQL
docker run -d --name pg-scheduler \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=job_scheduler \
-p 5432:5432 postgres:16-alpine
# 2. Backend (env vars use APP_ prefix — see .env.example)export APP_DATABASE_URL=postgres://postgres:postgres@localhost:5432/job_scheduler
cargo run --bin api-server # API on :3000
cargo run --bin worker # Worker process
cargo run --bin scheduler-daemon # Scheduler singleton# 3. Frontendcd frontend
npm install
npm run dev # Vite dev server on :5173, proxies /api → :3000

Seed Data

On first deployment, three database migrations are applied automatically:

Initial schema — all 14 tables, 6 enum types, indexes, triggers
Idempotency fix — replaces UNIQUE NULLS NOT DISTINCT with a partial unique index (so multiple jobs can have a NULL idempotency key)
Seed data — a complete set of sample data for exploration

Accounts (password: `password123`)

Email	Full Name	Role in Acme Corp
`admin@example.com`	Alice Admin	owner
`dev@example.com`	Bob Developer	admin
`viewer@example.com`	Carol Viewer	viewer

Projects & Queues

Org	Project	Queues
Acme Corp	Web App	`default` (×5), `high-priority` (×10, p=10), `email` (×3)
Acme Corp	Background Jobs	`default` (persistent retry, 10 attempts)
Startup Inc	Analytics Pipeline	`analytics` (×2)

Jobs (10 across 7 statuses)

Status	Count	Examples
completed	3	noop, sleep_100
queued	1	sleep_1000 — waiting for a worker
scheduled	1	welcome email — runs in 2 hours
running	1	sleep_5000 — in-flight
failed	2	Connection refused, Payment timeout
dead_letter	1	Exhausted 5 attempts: downstream 500
cancelled	1	Cancelled by user

Plus: 11 execution records, 7 log lines, 2 DLQ entries (one from max attempts, one from lease expiration), 4 cron scheduled jobs (daily cleanup, hourly digest, weekly report, inactive job), 3 job batches (100-job nightly, 500-job import, 50-job migration).

Architecture

System Topology

graph TB
subgraph "Client Layer"
Browser["Browser<br/>(React SPA)"]
API_Client["External API Client<br/>(JWT / API Key)"]
end
subgraph "Gateway"
Nginx["Nginx<br/>(SPA + reverse proxy)"]
end
subgraph "Application Layer"
API["api-server<br/>(Axum HTTP + WS)<br/>N replicas, stateless"]
Worker["worker<br/>(poll + claim + execute)<br/>N replicas"]
Scheduler["scheduler-daemon<br/>(advisory-lock singleton)<br/>1 active + 1 standby"]
end
subgraph "Data Layer"
PG[("PostgreSQL 16<br/>jobs · queues · workers<br/>executions · logs · DLQ")]
end
subgraph "Observability"
Prom["Prometheus<br/>/metrics scraping"]
Grafana["Grafana<br/>dashboards"]
end
Browser --> Nginx
API_Client --> Nginx
Nginx -->|"REST + WebSocket"| API
Worker -->|"poll + claim<br/>LISTEN/NOTIFY"| PG
Scheduler -->|"promote · cron · reap"| PG
API -->|"queries"| PG
PG --> Prom
API --> Prom
Worker --> Prom
Scheduler --> Prom
Prom --> Grafana

Hexagonal (Ports & Adapters) Architecture

The backend is a modular monolith compiled as three independent binaries from a single Cargo workspace. Each binary scales independently in production without being separate repos.

graph TB
subgraph "bin/"
API_BIN["api-server / main.rs"]
WORKER_BIN["worker / main.rs"]
SCHED_BIN["scheduler-daemon / main.rs"]
end
subgraph "crates/ — Application"
API_CRATE["api<br/>Axum routes · middleware · DTOs<br/>error.rs · validation.rs"]
WORKER_RT["worker-runtime<br/>executor · heartbeat · shutdown"]
end
subgraph "crates/ — Domain Logic"
SCHED_CORE["scheduler-core<br/>claim · retry · cron · reaper"]
end
subgraph "crates/ — Domain (zero I/O)"
DOMAIN["domain<br/>entities · value objects · ports (traits)"]
end
subgraph "crates/ — Infrastructure"
INFRA["infra-postgres<br/>SQLx repository adapters"]
COMMON["common<br/>config · telemetry · errors"]
end
subgraph "External"
PG[("PostgreSQL")]
end
API_BIN --> API_CRATE
WORKER_BIN --> WORKER_RT
SCHED_BIN --> SCHED_CORE
API_CRATE --> DOMAIN
API_CRATE --> INFRA
WORKER_RT --> DOMAIN
WORKER_RT --> INFRA
SCHED_CORE --> DOMAIN
DOMAIN -. "defines traits" .-> INFRA
INFRA --> PG
COMMON --> DOMAIN

Dependency rule (enforced by Cargo.toml):

domain depends on nothing in the workspace — zero I/O dependencies
infra-postgres and scheduler-core depend on domain only
api and worker-runtime depend on domain + scheduler-core + infra-postgres — never on each other
Binaries are thin entry points wiring everything together

Claim Flow (the core reliability guarantee)

sequenceDiagram
participant W1 as Worker 1
participant W2 as Worker 2
participant PG as PostgreSQL
W1->>PG: BEGIN
W2->>PG: BEGIN
Note over W1,PG: CTE: SELECT id FROM jobs<br/>WHERE status='queued'<br/>ORDER BY priority DESC, run_at ASC<br/>FOR UPDATE SKIP LOCKED<br/>LIMIT free_permits
W1->>PG: Locks rows 1,2,3 (SKIP LOCKED)
W2->>PG: Locks rows 4,5 (skips 1,2,3)
W1->>PG: UPDATE SET status='claimed', locked_by=W1<br/>RETURNING *
W2->>PG: UPDATE SET status='claimed', locked_by=W2<br/>RETURNING *
W1->>PG: COMMIT
W2->>PG: COMMIT
Note over W1,W2: Zero contention, zero duplicate execution

Workers never SELECT then UPDATE separately — that's a race condition. The CTE with FOR UPDATE SKIP LOCKED is a single atomic operation.

Database Design

Entity-Relationship Diagram

erDiagram
users ||--o{ organization_members : "belongs to"
organizations ||--o{ organization_members : "has"
organizations ||--o{ projects : "owns"
projects ||--o{ queues : "contains"
projects ||--o{ retry_policies : "defines"
projects ||--o{ job_batches : "groups"
queues ||--o{ jobs : "holds"
queues ||--o{ scheduled_jobs : "templates"
queues ||--o{ dead_letter_queue : "failed"
retry_policies ||--o| queues : "default for"
retry_policies ||--o| jobs : "overrides"
retry_policies ||--o| scheduled_jobs : "applies to"
workers ||--o{ worker_heartbeats : "emits"
workers ||--o{ jobs : "locks"
workers ||--o{ job_executions : "executes"
jobs ||--o{ job_executions : "has attempts"
jobs ||--o{ job_logs : "produces"
jobs ||--o{ job_dependencies : "depends on"
jobs ||--o| dead_letter_queue : "dead-letters to"
job_executions ||--o{ job_logs : "contains"
job_batches ||--o{ jobs : "batches"
scheduled_jobs ||--o{ jobs : "spawns"
users {
uuid id PK
text email UK
text password_hash
text full_name
boolean is_active
timestamptz created_at
timestamptz updated_at
}
organizations {
uuid id PK
text name
text slug UK
timestamptz created_at
}
organization_members {
uuid org_id PK_FK
uuid user_id PK_FK
org_role role
}
projects {
uuid id PK
uuid org_id FK
text name
text api_key_hash UK
uuid created_by FK
timestamptz created_at
timestamptz updated_at
}
queues {
uuid id PK
uuid project_id FK
text name
int priority
int max_concurrency
boolean is_paused
uuid default_retry_policy_id FK
int visibility_timeout_secs
timestamptz created_at
timestamptz updated_at
}
retry_policies {
uuid id PK
uuid project_id FK
text name
retry_strategy strategy
bigint base_delay_ms
float multiplier
bigint max_delay_ms
int max_attempts
boolean jitter
timestamptz created_at
}
jobs {
uuid id PK
uuid queue_id FK
uuid scheduled_job_id FK
uuid batch_id FK
text job_type
jsonb payload
int priority
job_status status
timestamptz run_at
text idempotency_key
uuid retry_policy_id FK
int max_attempts_override
int attempt_count
uuid locked_by FK
timestamptz locked_at
timestamptz lease_expires_at
jsonb result
text error_message
text trace_id
timestamptz created_at
timestamptz updated_at
timestamptz completed_at
}
job_executions {
uuid id PK
uuid job_id FK
uuid worker_id FK
int attempt_number
execution_status status
timestamptz started_at
timestamptz finished_at
bigint duration_ms
text error_message
text error_stack
jsonb result
timestamptz created_at
}
job_logs {
bigserial id PK
uuid execution_id FK
uuid job_id FK
timestamptz ts
text level
text message
}
workers {
uuid id PK
text hostname
int pid
text version
uuid[] subscribed_queues
int max_concurrency
worker_status status
timestamptz started_at
timestamptz last_seen_at
}
worker_heartbeats {
bigserial id PK
uuid worker_id FK
timestamptz ts
int active_job_count
float cpu_percent
float memory_mb
jsonb metadata
}
dead_letter_queue {
uuid id PK
uuid job_id FK
uuid queue_id FK
jsonb payload_snapshot
text failure_reason
int attempt_count
text last_error
timestamptz moved_at
boolean resolved
uuid resolved_by FK
timestamptz resolved_at
text resolution_action
}
scheduled_jobs {
uuid id PK
uuid queue_id FK
text name
text cron_expression
text timezone
text job_type
jsonb payload_template
uuid retry_policy_id FK
boolean is_active
timestamptz next_run_at
timestamptz last_run_at
timestamptz created_at
timestamptz updated_at
}
job_batches {
uuid id PK
uuid project_id FK
text name
int total_jobs
int completed_jobs
int failed_jobs
batch_status status
timestamptz created_at
timestamptz updated_at
}
job_dependencies {
uuid job_id PK_FK
uuid depends_on_job_id PK_FK
}

Key Indexes

Table	Index	Purpose
`jobs`	`(queue_id, status, priority DESC, run_at) WHERE status = 'queued'`	Claim query access path — the most important index in the schema
`jobs`	`(status, run_at) WHERE status = 'scheduled'`	Scheduler promotion sweep
`jobs`	`(lease_expires_at) WHERE status IN ('claimed','running')`	Reaper expired-lease sweep
`jobs`	`(queue_id, idempotency_key) WHERE idempotency_key IS NOT NULL`	Idempotent submission (partial unique)
`job_executions`	`(job_id, attempt_number)`	Retry history per job
`job_logs`	`(job_id, ts)`	Log tail per job
`worker_heartbeats`	`(worker_id, ts DESC)`	Staleness detection

Enums

graph LR
subgraph "job_status"
scheduled --> queued
queued --> claimed
claimed --> running
running --> completed
running --> failed
failed --> scheduled
failed --> dead_letter
scheduled --> dead_letter
claimed --> dead_letter
running --> dead_letter
queued --> cancelled
scheduled --> cancelled
end

Six enum types: job_status, execution_status, retry_strategy, worker_status, batch_status, org_role.

Job Lifecycle

Full State Machine

stateDiagram-v2
[*] --> scheduled : Job created with<br/>run_at in the future
[*] --> queued : Job created with<br/>run_at ≤ now()
scheduled --> queued : scheduler-daemon<br/>promotion sweep
queued --> claimed : worker atomically claims<br/>(FOR UPDATE SKIP LOCKED)
claimed --> running : worker begins execution
running --> completed : handler returns Ok
running --> failed : handler returns Err
failed --> scheduled : attempt_count < max_attempts<br/>(retry with backoff)
failed --> dead_letter : attempt_count ≥ max_attempts
scheduled --> dead_letter : lease expired + attempts exhausted
claimed --> scheduled : lease expired + attempts remain
running --> scheduled : lease expired + attempts remain
queued --> cancelled : user cancels
scheduled --> cancelled : user cancels
completed --> [*]
dead_letter --> [*]
cancelled --> [*]
note right of failed
Retry backoff reuses the
"scheduled" state.
attempt_count > 0
distinguishes retries.
end note

Retry Strategies

Strategy	Formula	Jitter
`fixed`	`delay = base_delay_ms`	`delay × uniform(0.5, 1.5)`
`linear`	`delay = base_delay_ms × attempt_count`	`delay × uniform(0.5, 1.5)`
`exponential`	`delay = min(base_delay_ms × multiplier^(attempt_count−1), max_delay_ms)`	`delay × uniform(0.5, 1.5)`

Lease & Reaper

Every claimed job gets a lease_expires_at = now() + visibility_timeout_secs
Healthy workers renew leases via the heartbeat ticker (every 10s)
scheduler-daemon's reaper sweeps expired leases every 15s:
- attempt_count < max_attempts → reschedule for retry
- attempt_count ≥ max_attempts → dead letter

API Overview

Authentication

Method	Path	Auth	Description
`POST`	`/api/auth/register`	Public	Register + get JWT tokens
`POST`	`/api/auth/login`	Public	Login + get JWT tokens
`POST`	`/api/auth/refresh`	Public	Rotate tokens with refresh token

Protected routes require Authorization: Bearer <token> header. JWT claims contain sub (user UUID), email, exp, iat.

Organizations & Projects

Method	Path	Description
`POST`	`/api/organizations`	Create org (user becomes owner)
`GET`	`/api/organizations`	List orgs for authenticated user
`POST`	`/api/projects`	Create project (returns API key once)
`GET`	`/api/projects?org_id=`	List projects
`GET`	`/api/projects/{id}`	Get project
`DELETE`	`/api/projects/{id}`	Delete project

Queues

Method	Path	Description
`POST`	`/api/queues`	Create queue
`GET`	`/api/queues?project_id=`	List queues with depth
`GET`	`/api/queues/{id}`	Get queue details + depth
`DELETE`	`/api/queues/{id}`	Delete queue
`POST`	`/api/queues/{id}/pause`	Pause processing
`POST`	`/api/queues/{id}/resume`	Resume processing

Jobs

Method	Path	Description
`POST`	`/api/jobs`	Submit job (supports idempotency keys, delayed run_at, dependencies)
`GET`	`/api/jobs?queue_id=&status=&job_type=&limit=&offset=`	List jobs with filtering & pagination
`GET`	`/api/jobs/{id}`	Get job details
`POST`	`/api/jobs/{id}/cancel`	Cancel a job
`GET`	`/api/jobs/{id}/executions`	Get execution history
`GET`	`/api/jobs/{id}/logs`	Get job logs

Dead Letter Queue

Method	Path	Description
`GET`	`/api/dead-letter?queue_id=&resolved=`	List DLQ entries (no queue_id = all queues)
`POST`	`/api/dead-letter/{id}/resolve`	Resolve entry (`requeued` or `discarded`)

Retry Policies

Method	Path	Description
`POST`	`/api/retry-policies`	Create policy
`GET`	`/api/retry-policies?project_id=`	List policies for project
`GET`	`/api/retry-policies/{id}`	Get policy
`DELETE`	`/api/retry-policies/{id}`	Delete policy

Scheduled Jobs (Cron)

Method	Path	Description
`POST`	`/api/scheduled-jobs`	Schedule a cron job
`GET`	`/api/scheduled-jobs`	List active scheduled jobs
`GET`	`/api/scheduled-jobs/{id}`	Get scheduled job
`POST`	`/api/scheduled-jobs/{id}/deactivate`	Deactivate

Batches, Workers, Health & Metrics

Method	Path	Auth	Description
`GET`	`/api/batches/{id}`	JWT	Get batch progress
`GET`	`/api/workers`	JWT	List active workers
`GET`	`/healthz`	Public	Liveness (always 200)
`GET`	`/readyz`	Public	Readiness (checks DB)
`GET`	`/metrics`	Public	Prometheus metrics
`WS`	`/ws/events`	Public	WebSocket for live events

Error Response Format

All errors return a consistent JSON structure:

{
"error": "validation_error",
"message": "Validation failed",
"details": [
{"field": "email", "message": "Invalid email format"},
{"field": "password", "message": "Password must be at least 8 characters"}
]
}

Error codes: bad_request, unauthorized, forbidden, not_found, conflict, validation_error, too_many_requests, internal_error.

Frontend

A React 18 SPA served by nginx, with TanStack Query (server state), Zustand (auth store), Recharts (dashboard charts), and Tailwind CSS.

Pages

Page	Route	Features
Dashboard	`/`	Stats cards (queues, queued, running, DLQ, workers), Recharts bar chart (queue depth), pie chart (status distribution), cron job list, org selector
Queues	`/queues`	Project selector, create queue form, pause/resume, per-queue depth breakdown (queued/scheduled/running/DLQ), link to jobs
Jobs	`/jobs`	Status filter, paginated data table with status badges, execution history, job log viewer, cancel action
Workers	`/workers`	Live grid with status dots, auto-refresh every 5s, host/version/concurrency details
Dead Letter	`/dead-letter`	DLQ entries with requeue/discard, payload snapshot expand, failure reason + last error
Retry Policies	`/retry-policies`	Creator form, strategy badges (fixed/linear/exponential), delay configs
Scheduled Jobs	`/scheduled-jobs`	Cron job cards with next-run display, create/deactivate, queue selector
Login	`/login`	Login/register toggle, Zustand auth store integration, protected route redirect

Authentication Flow

User logs in → JWT access + refresh tokens stored in localStorage
All API calls attach Authorization: Bearer <token> header automatically
Protected pages redirect to /login when unauthenticated
Auth state managed via Zustand store (useAuthStore)

Observability

Metrics Catalogue

Metric	Type	Labels	Purpose
`jobs_created_total`	Counter	`queue`, `job_type`	Intake rate
`jobs_claimed_total`	Counter	`queue`, `worker_id`	Claim throughput
`jobs_completed_total`	Counter	`queue`, `job_type`	Success throughput
`jobs_failed_total`	Counter	`queue`, `job_type`, `terminal`	Failure rate
`job_execution_duration_seconds`	Histogram	`queue`, `job_type`	p50/p95/p99 latency
`queue_depth`	Gauge	`queue`, `status`	Backlog per status
`worker_active_jobs`	Gauge	`worker_id`	Concurrency utilization
`worker_up`	Gauge	`worker_id`	Worker health
`dlq_size`	Gauge	`queue`	DLQ growth
`lease_expirations_total`	Counter	`queue`	Crashed worker rate

Logging

All three binaries emit structured JSON logs (configurable to pretty-format for development). Every log line carries: timestamp, level, target (module path), span fields (job_id, queue_id, worker_id, attempt), thread.id, thread.name.

Example:

{
"timestamp": "2026-07-04T12:00:00.000Z",
"level": "INFO",
"target": "worker_runtime::executor",
"fields": {
"job_id": "a1b2c3d4-...",
"queue_id": "e5f6a7b8-...",
"attempt": 1,
"duration_ms": 234
},
"message": "Job completed successfully"
}

Project Structure

├── Cargo.toml # Workspace root
├── .env.example # All configuration (APP_ prefixed)
├── docker-compose.yml # Full stack: PG + 3 binaries + frontend + Prom + Grafana
│
├── bin/
│ ├── api-server/main.rs # Thin: metrics init + axum::serve
│ ├── worker/main.rs # Thin: queue discovery + poll loop + graceful shutdown
│ └── scheduler-daemon/main.rs # Thin: advisory lock + promotion/cron/reaper loops
│
├── crates/
│ ├── domain/ # Entities, value objects, ports (ZERO I/O)
│ │ └── src/
│ │ ├── entities.rs # 14 entity structs
│ │ ├── value_objects.rs # 6 enums (JobStatus, RetryStrategy, …)
│ │ ├── ports.rs # 11 async repository traits
│ │ └── errors.rs # DomainError enum
│ │
│ ├── common/ # Shared infrastructure
│ │ └── src/
│ │ ├── config.rs # Env-var config (APP_ prefix via figment)
│ │ ├── telemetry.rs # tracing-subscriber + Prometheus init
│ │ └── error.rs # CommonError type
│ │
│ ├── infra-postgres/ # SQLx adapters (implements domain::ports)
│ │ └── src/
│ │ ├── repositories/ # 11 repository impls
│ │ │ ├── job.rs # ★ Claim query (SKIP LOCKED CTE)
│ │ │ ├── queue.rs # CRUD + depth (Uuid::nil = all)
│ │ │ ├── worker.rs # Registration + heartbeats
│ │ │ ├── execution.rs # Execution records + logs
│ │ │ ├── dead_letter.rs # DLQ (Option<Uuid> = all queues)
│ │ │ ├── scheduled_job.rs # Cron template management
│ │ │ ├── user.rs # Registration + lookup
│ │ │ ├── organization.rs # Org membership + RBAC
│ │ │ ├── project.rs # Project CRUD
│ │ │ ├── retry_policy.rs # Retry policy CRUD
│ │ │ └── batch.rs # Batch progress tracking
│ │ └── migrations.rs # sqlx::migrate! runner
│ │
│ ├── scheduler-core/ # Pure business logic
│ │ └── src/
│ │ ├── claim.rs # JobClaimer
│ │ ├── retry.rs # RetryCalculator (pure fn, unit tested)
│ │ ├── cron_eval.rs # CronEvaluator (parse + next run)
│ │ └── lease_reaper.rs # LeaseReaper (sweep + DLQ insert)
│ │
│ ├── api/ # Axum HTTP layer
│ │ └── src/
│ │ ├── lib.rs # AppState + build_router()
│ │ ├── dto.rs # Request/response DTOs
│ │ ├── error.rs # AppError (8 variants → HTTP status)
│ │ ├── validation.rs # Validator (email, password, bounds, enums)
│ │ ├── middleware/
│ │ │ ├── auth.rs # JWT decode + Claims → Extension
│ │ │ ├── request_id.rs # UUID X-Request-Id
│ │ │ ├── rate_limit.rs # Token bucket limiter (bonus)
│ │ │ ├── api_key.rs # X-API-Key auth (bonus)
│ │ │ └── rbac.rs # Org role checking (bonus)
│ │ ├── routes/
│ │ │ ├── auth.rs # register, login, refresh
│ │ │ ├── health.rs # GET /healthz, /readyz
│ │ │ ├── metrics.rs # GET /metrics (global handle)
│ │ │ ├── projects.rs # CRUD /api/projects
│ │ │ ├── queues.rs # CRUD + pause/resume
│ │ │ ├── jobs.rs # CRUD + cancel + executions + logs
│ │ │ ├── workers.rs # GET /api/workers
│ │ │ ├── organizations.rs # CRUD /api/organizations (real JWT user)
│ │ │ ├── dead_letter.rs # List + resolve
│ │ │ ├── retry_policies.rs # CRUD /api/retry-policies
│ │ │ ├── scheduled_jobs.rs # CRUD + deactivate
│ │ │ └── batches.rs # GET /api/batches/{id}
│ │ └── websocket.rs # WS /ws/events
│ │
│ └── worker-runtime/ # Job execution engine
│ └── src/
│ ├── lib.rs # WorkerConfig + QueueRuntime
│ ├── executor.rs # poll + claim + execute + retry/DLQ
│ ├── heartbeat.rs # HeartbeatService (ticker + lease renew)
│ ├── shutdown.rs # GracefulShutdown (SIGTERM → drain)
│ └── job_handlers.rs # HandlerRegistry + noop/sleep/failing handlers
│
├── migrations/
│ ├── 20240704000001_initial_schema.up.sql # Full schema
│ ├── 20240704000002_fix_idempotency.up.sql # Partial unique index
│ └── 20240704000003_seed_data.up.sql # Sample users/orgs/jobs/DLQ/cron
│
├── docker/
│ ├── Dockerfile.backend # Multi-stage (cargo-chef, --release)
│ ├── Dockerfile.frontend # Multi-stage (Vite → nginx)
│ ├── nginx.conf # SPA fallback + /api + /ws proxy
│ ├── prometheus.yml # Scrape config for all 3 binaries
│ └── grafana/provisioning/ # Datasource auto-provisioning
│
└── frontend/
├── package.json
├── vite.config.ts
├── tailwind.config.js
└── src/
├── main.tsx # React root + QueryClient + Router
├── App.tsx # Navigation + 7 routes + sign out
├── api-client/index.ts # Typed API client (all endpoints)
├── stores/auth.ts # Zustand auth store
├── types/index.ts # TypeScript interfaces
└── pages/
├── Dashboard.tsx # Recharts + stats + cron list
├── Queues.tsx # Create/pause/resume + depth cards
├── Jobs.tsx # Filters + table + executions + logs
├── Workers.tsx # Live grid with status dots
├── DeadLetter.tsx # Requeue/discard + payload view
├── RetryPolicies.tsx # Strategy badges + create form
├── ScheduledJobs.tsx # Cron cards + create/deactivate
└── Login.tsx # Login/register toggle

Configuration

All configuration via environment variables with APP_ prefix (see .env.example):

Variable	Default	Description
`APP_DATABASE_URL`	`postgres://postgres:postgres@localhost:5432/job_scheduler`	Connection string
`APP_DATABASE_MAX_CONNECTIONS`	`10`	Pool max (per binary)
`APP_JWT_SECRET`	`change-me-in-production`	HMAC signing key
`APP_JWT_ACCESS_EXPIRY_SECS`	`900`	Access token TTL
`APP_JWT_REFRESH_EXPIRY_SECS`	`604800`	Refresh token TTL
`APP_API_HOST`	`0.0.0.0`	Bind address
`APP_API_PORT`	`3000`	HTTP port
`APP_METRICS_PORT`	`9001`	Prometheus port
`APP_WORKER_POLL_INTERVAL_MS`	`1000`	Poll interval
`APP_WORKER_MAX_CONCURRENCY`	`10`	Per-worker cap
`APP_WORKER_HEARTBEAT_INTERVAL_SECS`	`10`	Heartbeat interval
`APP_WORKER_GRACEFUL_SHUTDOWN_TIMEOUT_SECS`	`30`	Drain timeout
`APP_SCHEDULER_POLL_INTERVAL_MS`	`2000`	Promotion interval
`APP_SCHEDULER_LEASE_REAPER_INTERVAL_SECS`	`15`	Reaper interval
`APP_SCHEDULER_ADVISORY_LOCK_ID`	`42`	Singleton lock key
`APP_LOG_LEVEL`	`info`	Tracing filter
`APP_LOG_FORMAT`	`json`	`json` or `pretty`

Design Decisions

Postgres as the Queue (not Kafka/RabbitMQ)

Pro	Con
One fewer moving part	Lower raw throughput than dedicated broker
`SKIP LOCKED` for lock-free consumers	Queue depth growth increases claim cost
`LISTEN/NOTIFY` for event-driven wake-up	Polling is the safety net
Transactional create + enqueue in one write	Hot-table vacuum needed at scale

For throughput up to ~10k jobs/sec, Postgres-as-queue is more than sufficient and operationally simpler.

Retry Backoff Reuses 'scheduled' State

Rather than introducing a separate retrying state, a job waiting for its initial run_at and a job waiting for its next retry are the same: "not ready yet." attempt_count > 0 distinguishes retries.

Idempotency: At-Least-Once + Idempotent Handlers

The system guarantees at-least-once delivery. Exactly-once is provably impossible in a distributed system. The correct target is at-least-once with idempotent job handlers.

Denormalization Trade-offs

job_logs.job_id — avoids a JOIN on the highest-write table for "tail this job's logs"
dead_letter_queue.payload_snapshot — DLQ entries survive job purges/archives

Known Future Work

job_logs partitioning — monthly range with retention-based DROP
Soft deletes on projects and queues
Job archiving — move old completed/dead_letter jobs to jobs_archive
Redis rate limiting — cross-replica token buckets
Workflow dependencies — job_dependencies table exists; promotion sweep should check satisfaction

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
bin		bin
crates		crates
docker		docker
frontend		frontend
migrations		migrations
.env.example		.env.example
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
README.md		README.md
docker-compose.yml		docker-compose.yml

Folders and files

Latest commit

History

Repository files navigation