fix(collector): make Polymarket parity verification append-tolerant - #753
Conversation
The shadow parity validator read every tape in both spools twice and required each tape fingerprint (size and mtime included) to stay bit-identical across both passes. The gate keeps the baseline lane collecting by contract, so on high-rate tapes an append or rotation lands mid-pass with probability approaching 1, the fixed retry budget exhausts, and the verifier bails with "spool changed while reading parity window", failing the gate closed with no parity evidence at all. Tapes are append-only, so treat growth as safe: snapshot (device, inode, size) at pass start, cap the read at the snapshot size, and afterwards verify only that identity held and the tape never shrank. An in-flight partial final row is ignored on the live active tape but stays fail-closed on closed tapes. Truncation, replacement, rename, and symlink indirection still fail or retry exactly as before. Closed segments whose rotation stamp predates started_at - 1h cannot contain comparison rows (trades and settlements are window-bound, and metadata enters the projection at most 900 s before the window while discovery only fetches markets ending within 30 minutes of retrieval), so they are skipped entirely and read cost now tracks the comparison window instead of spool retention. The monday.polymarket_shadow_parity.v1 evidence schema and every gate predicate are unchanged; fail-closed behavior for sequence gaps, in-window schema violations, and duplicate record ids is covered by unchanged existing tests plus new truncation/replacement/symlink regressions. Refs #747
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 59 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Change contract
Make the Polymarket shadow parity validator able to verify high-rate tapes: the validator's read model was "read the entire spool twice and require the world not to move" while the gate contract keeps the baseline lane collecting, so on high-rate tapes the stable-read race exhausted the retry budget and the verifier bailed with
spool changed while reading parity window, failing the gate closed with no parity evidence (operational block, not a correctness finding). The fix changes what is read and how stability is judged:TapeSnapshotcaptures(device, inode, size)at pass start, the read is capped at the snapshot size, and afterwards only identity (same device/inode, no shrink) is re-verified. An in-flight partial final row is ignored on the live active tape but stays fail-closed on closed tapes. Truncation, replacement, rename, and symlink indirection keep failing/ retrying exactly as before.started_at - TAPE_WINDOW_LOOKBACK_SECONDS(1h) cannot contain comparison rows — trades and settlements are admitted only near the window, and metadata enters the projection at most 900 s before the window start while collector discovery only fetches markets ending within 30 minutes of retrieval — so they are never parsed. Read cost now tracks the comparison window, not spool retention.Analysis and acceptance criteria:
docs/reports/2026-08-08-polymarket-parity-high-rate-tapes-known-issue.md(items 1 and 2 of the proposed fix direction; items 3 and 4 — relaxed post-cutoff schema validity and bounded-memory streaming comparison — are deliberately left out as separate shippable behavior).Issue relationship
Refs #747
Out of scope
deployment/aliyun/polymarket-shadow-gate-policy.jqpredicates are untouched and themonday.polymarket_shadow_parity.v1evidence schema is unchanged.polymarket_raw.rs); no runtime or host mutation.Dependencies and merge order
None. Branched from
origin/mainb02c1ddf; written after #743 (7b8a61a6, Polymarket trade-row dedup) and verified orthogonal to it — that change is collector-internal poll dedup and does not alter tape format or validator semantics.Focused validation
From
rust_hft/:cargo test -p hft-collector --locked— 311 lib + 72 bin + integration tests, 0 failed.cargo clippy -p hft-collector --locked --all-targets— no lints (one pre-existing workspace-levelsqlxpatch notice, unrelated).New tests in
polymarket_parity.rs, each shown red against the pre-fix logic where applicable:concurrent_appends_to_the_live_legacy_tape_do_not_block_parity— a writer appends valid rows to the live legacy active tape at ≥100 rows/s whilecompare()runs over a 50k-row in-lookback segment; verification succeeds within the existing 5-attempt budget with passing evidence. Red on old code with the production failure mode (spool changed while reading parity window/ enumeration variants).closed_segments_outside_the_window_lookback_are_never_parsed— a retained segment rotated long before the window contains unparseable garbage; a passing comparison deterministically proves it is never read (any read is a hard error).appends_after_the_snapshot_are_excluded_not_retried— mid-pass growth under the same identity neither aborts nor is read by the pass (red on old code: fingerprint mismatch retry).mid_read_truncation_is_retried_instead_of_compared,replaced_tape_identity_between_passes_fails_closed,a_partial_trailing_row_on_a_closed_tape_fails_closed,symlinked_tape_still_fails_closed; sequence gaps, in-window schema violations, and duplicaterecord_idremain covered by the unchanged existing tests (out_of_window_rows_still_require_sequence_integrity,malformed_active_rust_trade_fails_before_rotation,duplicate_rust_trade_fails_dedupe_parity, …) — all passing.Counterexample boundary: a same-inode, same-size in-place rewrite with preserved mtime is not detected (as before, mtime equality was the only guard and it is gone); tapes are append-only by collector contract and replacement/rename/truncation are all still caught by device/inode/shrink checks plus directory re-enumeration between passes.
Environment note:
.github/scripts/agent-worktree-preflight.sh checkcurrently aborts with SIGPIPE (141) on this repo —git worktree list --porcelain | awk … exitunderpipefailwith 130+ registered worktrees; its exact checks were replicated manually for this worktree (verdict=ok, baseb02c1ddf). Not fixed here (outside this PR's file scope).Rollout and rollback
None. The verifier runs inside the shadow gate on already-collected evidence; this change only makes previously unattainable evidence producible. Rollback is reverting this single-file commit; no state, schema, or configuration changes.
Scope exception
None (1 file, 291 insertions / 27 deletions).