Skip to content

fix(seal): a failed cache write is not a refresh - #367

Merged
LKSNDRTMLKV merged 1 commit into
mainfrom
fix/a-failed-cache-write-is-not-a-refresh
Sep 17, 2026
Merged

LKSNDRTMLKV merged 1 commit into
mainfrom
fix/a-failed-cache-write-is-not-a-refresh

Conversation

@LKSNDRTMLKV

Copy link
Copy Markdown
Member

Closes #362.

record incremented RefreshStats before attempting the write and then dropped the error on the floor. The counters therefore described what the pass intended rather than what the cache holds — and it is the cache a verdict is answered from. A pass whose every write failed still published a full, healthy pair of gauges while the cache silently stopped ageing.

The half that #365 made urgent

The issue's title has two clauses and the second is the sharper one: "leaves the stale copy the fail-closed rule exists to drop".

A territory that verified last week and fails to verify today gets an Unavailable entry. If that write fails, the territory keeps its previous Verified row. Before #365 that was a latent problem — the cache was only read at boot. #365 made the refresh republish into the running inspector by re-reading the cache, so the stale Verified row now flows straight into served verdicts, with nothing in unchecked naming it. The verdict answers notListed for a provider on the strength of a list the node knows it could not refresh.

So the fix I merged earlier today widened this hole, and closing it is part of this change rather than a follow-up.

What changed

record counts after the write. A failed put increments a new unwritten and returns; verified and unavailable are only reached when the row landed. Those two are documented as describing the cache — unavailable is "the width of every notListed this node gives until the next pass" — and that is now true rather than nearly true.

unwritten_territories names them, unbounded rather than capped. The ordinary cause is the database being briefly unavailable, which hits every territory in the pass at once, so a pass that could write nothing must be able to say so about all of them.

The publish step admits them. Each unwritten territory is removed from lists and pushed onto unchecked with a reason. Dropped from lists as well as added to unchecked deliberately: leaving it in both would let the stale copy answer the question while the verdict merely expressed doubt about it. Narrow and honest beats wide and wrong — the same trade the module header already makes.

A pass that could not write comes back in 15 minutes rather than 24 hours. #345's failure-taxonomy bullet distinguishes causes by how retryable they are: a timeout is worth retrying, a signature that does not verify is not, Germany's parser ceiling (#320) never will be until the parser changes. A failed write is a fourth kind and the most retryable of them — it is this node's own database. Fifteen minutes rather than seconds because a pass is thirty-odd external fetches, and hammering the Union because Postgres blinked turns one outage into two; the fetches are the expensive half and they are not what failed.

Tests

Two, against a RefusesToWrite store double, both confirmed to bite by reverting record to its previous shape:

  • a_write_that_fails_is_not_counted_as_a_refresh — a list that verifies perfectly, so the only thing that can go wrong is the write. verified must stay 0.
  • an_unavailable_entry_that_cannot_be_written_is_unwritten_not_unavailable — the fail-closed case: the row that would have replaced a Verified copy cannot be written, and counting it as unavailable would claim the cache had been narrowed when it had not.

Against the old code both fail with left: 1, right: 0 on exactly the count that is supposed to describe the cache's width.

Incidental

RefreshStats loses its Copy derive, because it now carries a Vec<String>. Nothing depended on copying it.

Not in this branch

The retry interval is a constant, not configurable. Same reasoning RENEWAL_LEAD records: a threshold becomes a policy the moment something acts on it, and there is no evidence yet about what interval an operator would want.

just check green.

@LKSNDRTMLKV
LKSNDRTMLKV merged commit f75d8e4 into main Sep 17, 2026
14 checks passed
@LKSNDRTMLKV
LKSNDRTMLKV deleted the fix/a-failed-cache-write-is-not-a-refresh branch September 17, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A failed trusted-list write is counted as a refresh, and leaves the stale copy the fail-closed rule exists to drop

1 participant