Skip to content

[Rust port] relayburn-ledger: JSONL + lock + sqlite archive #243

Description

@willwashburn

Parent:#240

Context

Port the append-only JSONL ledger, content sidecar, lock protocol, and SQLite archive to Rust. Highest-risk crate after the parsers because the lock protocol and SQLite path have correctness invariants the rest of the system depends on.

Scope

JSONL append + content sidecar

  • Port packages/ledger/src/file-adapter.ts and packages/ledger/src/index-sidecar.ts (the per-line JSON.parse loop at index-sidecar.ts:188-203).
  • Port the hot loop at packages/ledger/src/archive.ts:1237-1266 (createReadStream → JSON.parse per line) using serde_json::Deserializer::from_reader().into_iter().

Lock protocol — replicate behavior, not primitives

Do NOT use fs2::FileExt::lock_exclusive (flock(2)). It has different cross-process semantics on macOS/Windows and would silently change behavior. Replicate the TS protocol from packages/ledger/src/adapters/file-lock.ts:89-138 exactly:

  • Exclusive-file-creation (open(lp, 'wx') equivalent — Rust OpenOptions::new().write(true).create_new(true).open(lp)).
  • Two-phase retry: 50×20ms (1s fast) then 40×250ms (10s slow).
  • Orphan recovery: unlink lockfiles older than 5s when the held lock times out, then race the open again.
  • Re-entrancy on lock name via task-local storage equivalent to TS AsyncLocalStorage.
  • Lock names in use: archive, ledger-index, ledger, test-serialize.

SQLite archive

Port packages/ledger/src/archive.ts:700-914 using rusqlite. TS already wraps the entire tail in a single BEGIN/COMMIT — don't "fix" what isn't broken. Real perf wins:

  • Multi-row VALUES (...), (...) inserts to amortize statement overhead vs per-row prepared statements.
  • WAL mode + synchronous=NORMAL for the rebuild path.
  • Prepared-statement reuse across tables instead of per-table .run() loops.

Schema compatibility is non-negotiable. Existing archive.sqlite files in the wild must keep working without a forced rebuild — or ship a one-time migration in burn state rebuild archive --full.

Files

  • packages/ledger/src/file-adapter.ts
  • packages/ledger/src/index-sidecar.ts:188-203
  • packages/ledger/src/archive.ts:700-914 (sqlite write tx)
  • packages/ledger/src/archive.ts:1237-1266 (hot parse loop)
  • packages/ledger/src/adapters/file-lock.ts:89-138 (lock protocol — copy semantics)
  • packages/ledger/src/lock.ts (the withLock wrapper)
  • packages/ledger/src/schema.ts (LedgerLine, TurnLine, StampLine)

Depends on

#241, #242

Acceptance

  • cargo test -p relayburn-ledger green.
  • Lock-protocol property test: 100 concurrent withLock("archive", …) callers all serialize correctly with no missed acquisitions; orphan recovery test artificially stales a lockfile and confirms recovery.
  • Cross-process lock test: a Rust-held lock blocks a TS-held lock and vice versa (run TS adapter and Rust adapter against the same lockfile).
  • Existing archive.sqlite from a 1.5GB ledger opens in the Rust crate and answers queries without a rebuild.
  • burn state rebuild archive --full (via Rust) on a 1.5GB ledger completes in ≤10s on M-series silicon.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions