Design doc: embed Litestream replication for sqlite-rest by Copilot · Pull Request #121 · b4fun/sqlite-rest

Copilot · 2026-01-14T07:11:41Z

The issue requests a design for embedding Litestream’s Go library to add replication/restore to sqlite-rest without altering the REST API.

Design scope: Opt-in Litestream replication for the SQLite DB; restore before serve; continuous WAL/snapshot replication; logging and Prometheus metrics; fail-fast defaults (no degraded mode) and divergence handled by failing fast with operator-driven force-restore if needed.
Configuration surface: Prefer a Litestream YAML config via --replication-config (minimal sqlite-rest flags: enable/config path); detailed tuning (snapshot/retention/replicas) delegated to Litestream config; least-privilege S3 IAM notes; multi-replica refers to multiple remote destinations for the same DB, with initial scope single.
Lifecycle & failure handling: Restore pre-open, start replicator alongside HTTP/metrics/pprof, monitor error channels and cancel the process on replication errors/divergence (fail-fast).
Testing plan (future work): Flag parsing, file-backed replication integration, restore flow, and metrics assertions.

Example lifecycle sketch:

restoreIfNeeded(ctx, dbPath, restoreURL, restoreOpts)
db:=openDB(...)
replicator:=newReplicator(dbPath, replicaURL, tuneOpts)
goreplicator.Start(ctx)
gometricsServer.Start(ctx)
gopprofServer.Start(ctx)
server.Start(ctx.Done())
// On replicator error: log/metric and cancel ctx to fail fast

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

bcho

@copilot

bcho · 2026-01-14T07:26:24Z

+
+### High-level flow
+
+1. **Configuration** (new `ReplicationOptions`):


change to consume litestream configuration file here to minimal the deltas.

Updated to prefer a Litestream YAML config via --replication-config so sqlite-rest changes stay minimal. (f01868e)

bcho · 2026-01-14T07:27:32Z

+ - `--replication-restore-lag` (duration, default `0` meaning no lag allowed; used during startup restore decisions to tolerate a small amount of staleness between the local DB and the replica before forcing a restore).
+ - Env var mirrors for container use (e.g., `SQLITEREST_REPLICATION_ENABLED`, etc.).
+
+2. **Restore before serving**:


describe the behavior of diverging with remote backup

Added divergence behavior: default fail-fast on lineage mismatch with remote, operator can force-restore or allow degraded if they accept overwrite risk. (f01868e)

bcho · 2026-01-14T07:29:20Z

+
+## Open questions
+
+- Should we expose multiple replicas at launch or keep single-replica until requested?


multiple replicas of what? Please clarify

Clarified this means multiple remote destinations for the same SQLite DB via Litestream; initial scope remains single replica. (f01868e)

remove this question as we have the answer

Removed the answered question about multiple replicas; doc now just states initial scope is single replica with Litestream config controlling replicas. (d41a7dd)

Removed the answered question; doc reflects single-replica initial scope with Litestream config controlling destinations. (d41a7dd)

bcho · 2026-01-14T07:29:47Z

+## Open questions
+
+- Should we expose multiple replicas at launch or keep single-replica until requested?
+- How strict should startup be when replication is enabled but the remote is unreachable? **Recommendation:** fail fast by default to avoid running without configured durability, with an explicit `--replication-allow-degraded` escape hatch if operators need to accept the data-loss risk.


agree with this decision, but we should refine the cli flag experience.

Flag UX is now minimal: enable + config path, optional restore-from/allow-degraded; all other tuning stays in the Litestream config. (f01868e)

remove this question

Removed the resolved question; CLI UX stays minimal via enable + config path, optional restore-from. (d41a7dd)

Removed the resolved question per feedback. (d41a7dd)

bcho · 2026-01-14T07:30:15Z

+
+- Should we expose multiple replicas at launch or keep single-replica until requested?
+- How strict should startup be when replication is enabled but the remote is unreachable? **Recommendation:** fail fast by default to avoid running without configured durability, with an explicit `--replication-allow-degraded` escape hatch if operators need to accept the data-loss risk.
+- What are the sensible defaults for snapshot/retention to balance durability and cost?


defaults to disable this feature, delegate the detailed settings to litestream config

Documented that replication stays disabled by default and detailed settings are delegated to the Litestream config file. (f01868e)

remove this question

Removed the answered question; defaults remain disabled with detailed settings delegated to the Litestream config. (d41a7dd)

Removed the remaining question; open questions now only cover sensible defaults for snapshot/retention. (d41a7dd)

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

bcho

@copilot

bcho · 2026-01-14T07:38:20Z

+
+## Open questions
+
+- Should we expose multiple replicas at launch or keep single-replica until requested?


remove this question as we have the answer

bcho · 2026-01-14T07:38:25Z

+## Open questions
+
+- Should we expose multiple replicas at launch or keep single-replica until requested?
+- How strict should startup be when replication is enabled but the remote is unreachable? **Recommendation:** fail fast by default to avoid running without configured durability, with an explicit `--replication-allow-degraded` escape hatch if operators need to accept the data-loss risk.


remove this question

bcho · 2026-01-14T07:38:34Z

+
+- Should we expose multiple replicas at launch or keep single-replica until requested?
+- How strict should startup be when replication is enabled but the remote is unreachable? **Recommendation:** fail fast by default to avoid running without configured durability, with an explicit `--replication-allow-degraded` escape hatch if operators need to accept the data-loss risk.
+- What are the sensible defaults for snapshot/retention to balance durability and cost?


remove this question

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

bcho

@copilot

bcho · 2026-01-14T07:57:19Z

+1. **Configuration** (new `ReplicationOptions`):
+ - `--replication-enabled` (bool, default false).
+ - `--replication-config` (string, path to Litestream YAML config; preferred path to keep sqlite-rest changes minimal and delegate detailed tuning like snapshot/retention/replicas to Litestream).
+ - `--replication-restore-from` (optional override to restore from a different replica URL; if omitted, use the primary replica from the Litestream config).


remove --replication-restore-from flag

Removed the --replication-restore-from flag; minimal flags are now enable + config path. (61a3a12)

bcho · 2026-01-14T07:57:29Z

+ - `--replication-config` (string, path to Litestream YAML config; preferred path to keep sqlite-rest changes minimal and delegate detailed tuning like snapshot/retention/replicas to Litestream).
+ - `--replication-restore-from` (optional override to restore from a different replica URL; if omitted, use the primary replica from the Litestream config).
+ - Env var mirrors for container use (e.g., `SQLITEREST_REPLICATION_ENABLED`, `SQLITEREST_REPLICATION_CONFIG`, etc.).
+ - Recommended CLI UX: keep flags minimal (`--replication-enabled`, `--replication-config`, optional `--replication-restore-from`) and leave all other Litestream knobs to the config file.


drop this line

Dropped the UX line tied to the removed flag; doc reflects only enable + config path. (61a3a12)

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Initial plan
b057512

CopilotAI assigned Copilot and bcho Jan 14, 2026

Copilotstarted work on behalf of bcho January 14, 2026 07:12 View session

CopilotAIand others added 9 commits January 14, 2026 07:15

docs: add litestream replication design
e79df89
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: clarify replication startup default
913d09d
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: refine multi-replica config note
bbb612e
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: align replica flag naming
322f752
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: clarify replica flag scope and error handling
46325d5
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: add go syntax hint to lifecycle sketch
f16d065
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: expand restore config and error handling
8aaf3ae
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: note s3 perms and rename degraded flag
229cb9e
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: clarify restore lag and degraded handling
6afce55
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

CopilotAI changed the title ~~[WIP] Design document for embedding litestream for replication~~Design doc: embed Litestream replication for sqlite-restJan 14, 2026

CopilotAI requested a review from bcho January 14, 2026 07:22

Copilotfinished work on behalf of bcho January 14, 2026 07:22

bcho requested changes Jan 14, 2026
View reviewed changes

Copilotstarted work on behalf of bcho January 14, 2026 07:31 View session

docs: shift to litestream config and clarify divergence
f01868e
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

CopilotAI requested a review from bcho January 14, 2026 07:34

Copilotfinished work on behalf of bcho January 14, 2026 07:34

bcho reviewed Jan 14, 2026
View reviewed changes

Copilotstarted work on behalf of bcho January 14, 2026 07:39 View session

docs: simplify flags and fail-fast defaults
d41a7dd
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

CopilotAI requested a review from bcho January 14, 2026 07:41

Copilotfinished work on behalf of bcho January 14, 2026 07:41

bcho requested changes Jan 14, 2026
View reviewed changes

Copilotstarted work on behalf of bcho January 14, 2026 07:58 View session

docs: remove restore-from flag and tighten ux
61a3a12
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

CopilotAI requested a review from bcho January 14, 2026 07:59

Copilotfinished work on behalf of bcho January 14, 2026 07:59


		### High-level flow

		1. Configuration (new `ReplicationOptions`):


		## Open questions

		- Should we expose multiple replicas at launch or keep single-replica until requested?

Uh oh!

Conversation

CopilotAI commented Jan 14, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bcho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

bcho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bcho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CopilotAIJan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

CopilotAI commented Jan 14, 2026•
edited
Loading