Skip to content

perf(state): halve the state, compress it on every core, retry a mutation - #9

Merged
jshvn merged 1 commit into
mainfrom
josh/checkpoint-diet
Sep 15, 2026
Merged

jshvn merged 1 commit into
mainfrom
josh/checkpoint-diet

Conversation

@jshvn

@jshvn jshvn commented Sep 15, 2026

Copy link
Copy Markdown
Member

Why

Measured over 56 batches (runs 25-36): the checkpoint costs 91 s flat per batch, 12% of a run and 21% of a large-file run. The state is 1.04 GB, of which 57% is one column nothing reads. Compression is single-threaded on a 5-core runner. Separately, one transient failure on an upload fails the batch and loses the night, because mutations never retried.

Change

  • State trim. proton_nodes.raw_json and dropbox_objects.raw_json are written as {}: every field the pipeline reads is its own column, and the inventory already blanked the Dropbox one after the fact (that pass is gone). CHECKPOINTED batch items are pruned at plan time (already merged into mirror_objects; CONFIRM_FAILED rows stay for the operator). Events and commands older than seven days are pruned, keeping the reconcile figures events the report reads from the latest complete walk. Measured on the live state: 1,038 MB to 498 MB, the pushed object 140 MB to 96 MB.
  • Parallel compression. The snapshot compresses in 64 MB slices across a process pool, one xz stream each, written in order. xz permits concatenated streams and lzma reads them as one file, so the object format is unchanged, every history copy stays readable, and task state-rollback works in both directions. Measured 3.2x on 5 cores at identical size. The restore streams instead of holding the whole file in memory.
  • synchronous=NORMAL. WAL mode still survives a process crash; the canonical state is the R2 object fetched at the start of every run, so a runner's disk never outlives what it holds.
  • Staging is cleared before the push, not after: it is dead once confirm counted it, and this keeps the snapshot and its compressed copies off a disk still holding a whole batch.
  • Bounded mutation retry. proton.mutation_max_attempts (default 3) retries a timed-out or failed mutation after the existing backoff, never after an authentication failure. A repeated upload of identical bytes is skipped by the CLI and confirm's accounting still balances, so a retry can only cost time. ProtonCLIError carries the attempt's category.

Nothing about what is recorded as mirrored changes. The checkpoint's history-then-canonical ordering and per-batch cadence are unchanged.

Test plan

  • task test: 160 passed, five new: a multi-stream push restores byte for byte; a transient mutation failure is retried and an auth failure is not; checkpointed items are pruned and failed ones kept; old log rows go and reconcile figures stay; the listing stores no raw JSON
  • task lint, task check
  • After merge, watch checkpoint_seconds in batches.details_json fall from ~91 s and task status still read the state

@jshvn
jshvn merged commit b680c6d into main Sep 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant