PRD: Polymarket collector data integrity and release orchestration recovery

[proerror77]

Problem Statement

The production Polymarket lane cannot be promoted predictably even though its Rust binaries and immutable artifacts exist. The market recorder, reference collector, and uploaders are separate runtime products but have repeatedly been treated as one “Rust collector.” Real closed-hour data is structurally valid yet non-canonical because WebSocket reconnects expand into token-wide quote failures, late source deltas and crossed-book batches force reconnects, and events discovered after their end time are still counted as requiring quote coverage. The release path also relies on manual artifact staging and a long Cloud Assistant-owned Gate; an Assistant restart can terminate the formal invocation while leaving an orphan Gate process. Operators therefore lack one repository-owned, fail-closed path from exact source and signed artifacts to verified production services.

Solution

Provide separate, repository-owned release paths for the Polymarket market recorder and raw-ops services. Supervise the long raw-ops Gate independently from Cloud Assistant, stage immutable artifacts atomically, deploy and roll back the market recorder with exact binary identity checks, correct only proven market-data ordering and event-lifecycle completeness defects, and finish with new exact-main artifacts, a real closed-segment preflight, a complete 3600+601-second Gate, immutable evidence verification, and production cutover. Every failure preserves the active Python baseline or the last verified recorder and leaves Rust shadow inactive.

User Stories

1. As an operator, I want the market recorder and raw-ops services named and released separately, so that I know which binary and data contract is changing.
2. As an operator, I want a long Gate to survive Cloud Assistant restarts without becoming unmanaged, so that controller transport failures cannot create an orphan promotion process.
3. As an operator, I want every Gate to have one queryable runtime identity and terminal result, so that I can prove whether it passed, failed, or was cancelled.
4. As an operator, I want Gate cancellation to stop shadow and preserve Python, so that recovery remains fail-closed.
5. As an operator, I want exact remaining RuntimeMaxSec evidence before Gate admission, so that scheduled Python restarts cannot invalidate the observation window.
6. As an operator, I want raw-ops artifacts staged by one repository-owned command, so that the release manifest cannot be omitted from an otherwise valid control directory.
7. As an operator, I want staging to reject mixed-source binaries and controls, so that candidate identity remains immutable.
8. As an operator, I want staging to be atomic and non-overwriting, so that a partial install cannot become a release candidate.
9. As an operator, I want the market recorder installed from its signed artifact, so that /opt/monday/bin/new-ploy-runner is not replaced manually.
10. As an operator, I want recorder deployment to verify /proc/PID/exe SHA and source revision, so that service configuration alone cannot masquerade as deployment proof.
11. As an operator, I want recorder rollback to restore the prior verified binary and service identity, so that a bad rollout is recoverable.
12. As a data consumer, I want superseded same-token WebSocket deltas ignored only when newer state proves they are obsolete, so that late network delivery does not invalidate unrelated current data.
13. As a data consumer, I want malformed or unproven late deltas to remain fail-closed, so that ordering tolerance cannot silently corrupt L2 state.
14. As a data consumer, I want crossed-book failures reproduced from real payload evidence before behavior changes, so that the collector does not normalize away a genuine invalid book.
15. As a data consumer, I want independent entries in a mixed batch preserved when one entry is proven superseded, so that one token does not invalidate every subscription.
16. As a data consumer, I want events discovered after their end time excluded from active quote-coverage obligations, so that impossible quote attempts do not make an otherwise complete hour non-canonical.
17. As a data consumer, I want active events to retain strict quote and attempt coverage, so that lifecycle correction cannot weaken completeness.
18. As an auditor, I want failure batches and affected tokens reported separately, so that per-token expansion is not confused with the number of transport incidents.
19. As an auditor, I want real closed-hour validation to prove JSON, sequence, timestamps, quote status, price/size bounds, lifecycle coverage, and OSS triplet readback, so that “the file is still growing” is not mistaken for correctness.
20. As a release owner, I want exact-main artifacts rebuilt after every relevant merged data or control change, so that an older preflight cannot authorize a newer deployment.
21. As a release owner, I want a real market segment preflight before shadow, so that uploader compatibility is proven against production format.
22. As a release owner, I want the full reference Gate to remain canonical-positive, so that non-canonical compatibility acceptance cannot promote incomplete production data.
23. As a release owner, I want immutable gate.json and PASSED.sha256 bound to source, candidate, controls, observation window, parity, and OSS readback, so that cutover evidence is independently verifiable.
24. As an operator, I want final cutover to verify all three raw-ops ExecStart values, actual process SHA, timers, upload services, reference artifacts, market-tape triplets, and visible L2, so that repository merge is not confused with production completion.

Implementation Decisions

• Treat the market recorder and raw-ops release as separate rollout and rollback units.
• Reuse the existing market-recorder release manifest/verifier, raw-ops release manifest, control-plane Gate, cutover checks, and uploader closed-segment validator.
• Add only repository-owned commands required to stage, supervise, deploy, query, cancel, and roll back these existing artifacts; do not introduce a general deployment framework.
• Run the long Gate under systemd supervision with a stable unit identity and bounded lifetime. Cloud Assistant may start and inspect it but does not own its process tree.
• Persist terminal Gate status and immutable evidence before promotion. A missing or ambiguous terminal state is failure.
• Stage raw-ops binary, release manifest, control manifest/archive, and controls into one immutable destination before atomic publication.
• Install the recorder through an atomic verified replacement with a recoverable previous release; verify configured ExecStart and actual /proc/PID/exe identity after restart.
• Preserve strict rejection of malformed depth, invalid prices/sizes, crossed final books, unproven stale deltas, sequence gaps, and tampered OSS evidence.
• Correct event coverage using event lifecycle at discovery/observation time; only tokens belonging to events eligible for collection in the segment require quote attempts.
• Keep collector deployment, data-contract corrections, and final production cutover in separate issues and PRs.
• Keep ResearchSnapshot, evaluator/MCTS, ACK Job, Paper/Shadow/Live execution, issue Capture full-update Polymarket CLOB evidence for token microstructure research #235, and unrelated worktrees untouched.

Testing Decisions

• Use the existing raw-ops control-plane shell test as the highest seam for staging, Gate supervision, cancellation, identity, timeout, and fail-closed behavior.
• Use the existing market-recorder release contract test as the highest seam for artifact install, actual SHA verification, restart, and rollback.
• Use focused ploy-market-data tests with real counterexample shapes for stale and crossed WebSocket payload behavior.
• Use uploader validator tests containing active, expired-before-discovery, and mixed-lifecycle event tokens to prove canonical coverage semantics.
• Require targeted counterexamples: Cloud Assistant/controller disappearance with a running Gate; omitted manifest staging; wrong recorder SHA after restart; unproven stale delta; crossed final book; expired-at-discovery token; active token without an attempt.
• After code review and CI, verify the recorder using a new real closed hour and verify raw-ops using a new real closed-segment preflight and full production Gate.
• Treat production service identity, /proc/PID/exe SHA, immutable OSS triplets, and Gate receipts as separate required evidence.

Out of Scope

• ResearchSnapshot, evaluator, MCTS, ACK Job, Paper/Shadow/Live execution, and issue Capture full-update Polymarket CLOB evidence for token microstructure research #235.
• Weakening canonical data requirements to make a Gate pass.
• Rewriting the Python collector that will be retired.
• A generic deployment platform for other venues.
• Deleting old immutable releases or unrelated worktrees.
• Trading strategy or execution behavior.

Further Notes

The latest formal audit found a structurally valid one-hour tape with zero sequence gaps but 260 WebSocket failure batches expanded into 5,558 token failures. The active recorder source predated merged late-delta fixes. A formal Gate invocation was terminated by a Cloud Assistant restart while its process tree briefly remained alive; the orphan process was stopped, Rust shadow is inactive, and Python remains production. New rollout evidence must therefore come from final exact main and cannot reuse prior Gate invocations or artifacts.