local dev: sqlite queue for cross-process

[Curtis Summers (csummers)]

Managed SQLite queue for reliable local cross-process delivery

Summary

Some history: The default local development previously had a shared memory + IPC implementation for queues, but I eventually removed this in favor of a single-process memory-only queue implementation that worked well for hot dev. This change allowed for a simpler design, but gave up the ability to have hot eval 'send("my-event", ...)' run from another process in local development unless you were using the redis queue type locally. This PR fixes this with a sqlite queue + in-process message channel + out-of-process backup polling.

Adds a managed SQLite queue backend and makes it the default queue for Hot projects.

SQLite is the queue store. An in-process channel carries durable message IDs for immediate same-process wakeups, while a short SQLite polling interval discovers messages produced by sibling processes. This fixes event delivery between commands such as hot dev, hot run, and hot eval without requiring Redis.

Motivation

The previous in-memory default only worked when producers and consumers shared a process. Separate local commands could enqueue into isolated memory queues, causing events to be silently lost.

The new default provides:

• Reliable cross-process delivery in local projects
• Fast same-process notification without waiting for SQLite polling
• Durable standalone worker queues across process restarts
• No external queue service or application database migration requirements

Queue configuration

Introduces queue.type = "sqlite".

• sqlite: managed project-local queue and the new in-project default
• memory: explicitly selected, single-process-only queue
• redis: distributed/production queue
• none: queue disabled where supported

sqlite3 is not accepted as an alias.

Commands outside a Hot project continue to resolve the queue as disabled where appropriate.

Managed SQLite backend

• Stores queue databases under .hot/db/queue/
• Uses one managed database per named queue
• Owns its schema, tables, indexes, schema version, WAL settings, and lifecycle
• Remains independent from db.uri and application database migrations
• Uses atomic UPDATE ... RETURNING claims to prevent duplicate processing
• Renews leases while handlers are running
• Supports delayed retries, retry limits, and durable dead-letter storage
• Uses in-process message-ID notifications for low-latency delivery
• Polls SQLite every 50ms to discover cross-process messages
• Caps stale notification processing so authoritative database claims are not delayed
• Self-heals empty, partially initialized, or corrupt managed queue files
• Rejects unsupported future schema versions instead of interpreting them unsafely

Queue lifecycle semantics

Standalone services

Standalone hot api, hot app, hot worker, hot task-worker, and hot scheduler processes can share the SQLite queue.

Queue data survives process restarts, and expired processing leases are recovered for retry.

Redis-style startup retention APIs are intentionally non-destructive for SQLite. Startup windows and stale-backlog purges do not delete ready messages or another worker’s active lease because SQLite contains the authoritative copy of the work.

hot dev

hot dev preserves the session-scoped behavior of the former memory queue:

• Takes an exclusive project queue-session lock
• Snapshots managed queue messages at startup
• Reconciles interrupted Runs and Tasks before deleting messages
• Clears only rows present in the startup snapshot
• Preserves messages enqueued after the snapshot boundary
• Starts each dev session with a clean queue

Standalone SQLite services use shared queue-session locks and cannot overlap a running hot dev, preventing a dev restart from clearing their active work.

hot run and hot eval can still publish events into the active dev session.

hot test

Integration-mode hot test explicitly uses the memory queue because its services run in one process. This isolates tests from a sibling hot dev and avoids accumulating per-run SQLite files.

Recovery and dead-letter handling

Orphan recovery now distinguishes between:

• Messages requeued for another processing attempt
• Messages moved to the dead-letter queue after retry exhaustion

SQLite returns the durable payloads for both outcomes.

This allows:

• Event workers to mark affected Runs failed
• Task workers to mark affected Tasks failed and publish terminal events
• hot dev reconciliation to include dead-lettered event snapshots
• Periodic SQLite janitors to reconcile leases that expire after worker startup

Expired leases are no longer moved to the DLQ invisibly during an ordinary claim. Terminal transitions happen through explicit recovery, where the application state can be reconciled.

Dropped, cancelled, or panicking handlers consume paced retries rather than entering a tight redelivery loop.

Event publisher reliability

• Queue publisher constructors now return errors instead of panicking
• Fire-and-forget enqueue tasks are tracked
• Publisher shutdown waits for pending durable enqueues
• hot run and hot eval surface enqueue/drain failures to the user
• Execution and publisher failures are combined when both occur
• Dev shutdown drains publishers to reduce one-shot event loss

Queue administration

hot queue status supports managed SQLite queues and reports:

• Pending messages
• Processing messages
• Dead-letter messages

hot queue clear clears managed SQLite queue rows.

Documentation

Updates configuration documentation and the full configuration template with:

• The new SQLite default
• Available queue types
• Managed file location
• Standalone durability semantics
• hot dev reset behavior
• Queue-session locking
• Integration-test isolation

Compatibility

• Redis behavior remains unchanged
• Memory queues remain available by explicit configuration
• Existing queue.type = "redis" deployments continue to work
• The main behavior change is that projects without an explicit queue type now use managed SQLite instead of memory