PRD: make Polymarket BTC 5m research continuously ready and fast

[proerror77]

Problem Statement

Polymarket BTC five-minute research currently mixes data acquisition, data repair, evidence verification, snapshot construction, evaluator execution, MCTS search, CI image publication, and completion readback inside one human-driven serial loop. A hypothesis can wait for a fresh collector window, discover an old-schema incompatibility only in ACK, trigger unrelated CI lanes, or create another diagnostic Job before the evaluator starts.

The user needs a complete cloud-native Rust path where valid event data is continuously admitted before research, Up and Down retain independent instrument identity, settlement and execution use separate objectives, agents submit immutable missions against cached snapshots, and exact-main results are independently read back without rebuilding or reacquiring data.

Solution

Build one continuously ready Polymarket BTC 5m research platform from the existing Rust collector, OSS evidence, typed verifier, ResearchSnapshot, alpha-harness dispatcher, shared MCTS kernel, ACK runtime, and exact-main image pipeline.

A source-closed collector segment is verified at publication time and classified as ready, partial, or rejected per event. Ready events enter an immutable catalog and are materialized into authenticated common-time cohorts and cached ResearchSnapshots before Mission submission. Admission rejects unsupported targets or insufficient cohorts without creating a Pod.

The research contract exposes three independent objectives: market-level settlement probability, Up-token execution, and Down-token execution. A pipeline-smoke mode proves the technical path without an alpha claim; a research-trial mode enforces event-disjoint train/held-out cohorts and shared-MCTS isolation. CI selects only affected jobs, compiles an exact-main release artifact once, and reuses it for image publication.

The highest verification seam is:

Given an authenticated cohort identity, immutable Mission SHA, and exact image digest, the system starts from a verified cached ResearchSnapshot without compiling source, collecting data, or transferring the same snapshot again, enters the evaluator within 90 seconds on the warm path, and publishes an independently rehashed immutable result bundle.

User Stories

1. As a researcher, I want BTC 5m events to become research-ready continuously, so that a hypothesis does not wait for a new collector run.
2. As a researcher, I want one market_id to remain one episode, so that different binary-expiry products are never mixed.
3. As a researcher, I want Up and Down books to remain independent instruments, so that their liquidity and execution behavior are modeled correctly.
4. As a researcher, I want Chainlink stored once and causally projected by event, so that shared reference data neither duplicates storage nor leaks future observations.
5. As a researcher, I want settlement probability and token execution to be different objectives, so that final outcome is not confused with short-horizon tradability.
6. As a researcher, I want Up and Down execution evaluators to be independent, so that each token uses its own book, fills, fees, and slippage.
7. As a researcher, I want 5, 10, 15, and 30 second execution horizons to be explicit identities, so that horizons are never silently combined.
8. As a researcher, I want a pipeline-smoke mode, so that one complete event can prove compatibility without claiming generalization.
9. As a researcher, I want a research-trial mode, so that MCTS and held-out metrics only run on event-disjoint cohorts.
10. As a research agent, I want to submit an immutable Mission rather than edit code, so that parameter research does not create overlapping branches.
11. As a research agent, I want unsupported missions rejected before Pod creation, so that cloud time is not spent discovering known data gaps.
12. As a research agent, I want cached snapshots addressed by authenticated SHA, so that repeated hypotheses reuse the same bytes.
13. As a data owner, I want every event classified as ready, partial, or rejected with reason codes, so that historical objects are not repeatedly reinterpreted.
14. As a data owner, I want non-derivable provenance to fail closed, so that request status, source clocks, trades, and sequence evidence are never fabricated.
15. As a data owner, I want event-local completeness, so that unrelated gaps do not reject a complete selected event.
16. As an operator, I want deployed Rust binary and configuration identities recorded, so that repository and runtime state cannot drift silently.
17. As an operator, I want one typed dispatcher to render, dry-run, submit, observe, and read back a Mission, so that ad hoc Job wrappers are unnecessary.
18. As an operator, I want lifecycle timestamps, so that scheduling, image, snapshot, evaluator, MCTS, and publication latency are measurable.
19. As a developer, I want CI jobs selected by affected behavior, so that collector work does not run unrelated frontend and heavy-research lanes.
20. As a developer, I want relevant main changes to retain complete image validation, so that speed does not weaken the release contract.
21. As a developer, I want exact-main binaries compiled once and reused by ACR, so that one SHA is not compiled in multiple workflows.
22. As a developer, I want a current-schema producer-to-snapshot fixture, so that incompatibility is discovered in CI instead of ACK.
23. As a maintainer, I want strict WIP and ownership limits, so that scope control does not turn into dozens of diagnostic issues and branches.
24. As a risk owner, I want research completion separated from Paper, Shadow, Live, promotion, and profitability, so that E2E cannot enable trading authority.

Implementation Decisions

• The first delivered product family is BTC x 5m. Contracts retain explicit symbol and horizon identities for later products.
• Reuse the existing Rust collector, uploader, verifier, selector, ResearchSnapshot, dispatcher, snapshot cache, shared MCTS kernel, ACK Jobs, OSS, and ACR.
• Add no scheduler, message bus, database service, external authoritative sandbox, or custom operator unless measured evidence proves the existing seams insufficient.
• Data identity is product_family = symbol x horizon, episode = market_id, instrument = token_id.
• Every selected market binds exactly one Up and one Down token with independent order-book and trade surfaces.
• Store Chainlink once and causally project it by event window. Keep official settlement separate from repricing and execution.
• Classify publication as ready, partial, or rejected. Never synthesize trust-boundary fields that were not collected.
• Add an immutable ready-event catalog binding coverage, request outcomes, clocks, sequence state, producer/schema/verifier identity, supported tasks, and snapshot availability.
• Materialize snapshots before Mission execution. Missions consume authenticated snapshot identities, not raw collector paths.
• Mission v3 exposes product, typed task, run mode, authority profile, and cohort partition identity.
• Shared MCTS remains one kernel behind settlement, Up execution, and Down execution adapters.
• Pipeline smoke cannot produce an alpha verdict. Research trial uses event-disjoint common-time cohorts and held-out feedback cannot alter search.
• Binance surfaces are optional enhancements and never define Polymarket reference, settlement, books, or baseline admission.
• CI selects jobs at changed-path and Cargo dependency granularity. Relevant main changes retain full image validation.
• Relevant main CI builds one exact-SHA release artifact for both smoke and ACR packaging.
• Cloud writes retain one named controller. Branch, worktree, PR, Job, and artifact ownership are explicit.
• Diagnostic Jobs and readbacks remain evidence on the owning issue, not new rollout issues.

Testing Decisions

• Highest seam: submit an immutable Mission against a catalog-admitted cached Snapshot and verify evaluator start, result publication, and independent digest readback without build, collection, or repeated transfer.
• A three-event fixture crosses producer output, uploader, verifier, catalog, Snapshot, settlement Mission, Up Mission, Down Mission, and experiment result.
• Counterexamples cover missing request status, one-sided startup books, top/depth disagreement, sequence gaps, token mismatch, incomplete completion, future reference exposure, and unsupported tasks.
• Research tests prove separate task identities, 5/10/15/30 second horizons, event-disjoint partitions, crossing exclusion, and no held-out feedback in MCTS training.
• Dispatcher tests prove admission before Job creation, immutable identities, cache integrity, isolated concurrent workspaces, and no-clobber publication.
• CI selector tests assert exact job sets for a table of path and dependency changes.
• Release tests prove the smoke-tested and ACR-published images contain the same exact-main binary artifact.
• Runtime proof records cold, warm miss, warm hit, four concurrent Missions, and independent OSS readback.

Out of Scope

• Live execution, Paper or Shadow enablement, promotion, and profitability claims.
• Formal BTC 15m/1h, ETH, or SOL rollout; the contracts must permit them later.
• Full-update-tick microstructure training in the first baseline.
• Kafka, Redis, Argo, Fluid, ClickHouse migration, a custom operator, or external authoritative sandbox without a separate evidence-driven PRD.
• Reconstructing non-derivable historical trades, request outcomes, or source clocks.
• Unrelated CEX collection or trading-runtime authority.

Further Notes

• Acquire a bounded multi-event BTC 5m market cohort in ACK #313 remains suspended until typed acquisition preflight and the dependency graph are reconciled.
• Reuse PRD: make agent research loops start without rebuilds or repeated snapshot transfers #240 speed foundation and PRD: Verified Polymarket + Chainlink baseline research path #225 baseline work rather than rebuilding them.
• Reconcile BTC 5m cohort to sealed ResearchSnapshot #169, Make BTC 5m settlement probability a market-level research target #229, Add independent Up and Down token execution research tasks #230, Add a 15-second full-depth token repricing horizon #231, Run the exact-main Polymarket baseline evaluator and MCTS in cloud #233, Publish the Polymarket baseline E2E completion proof #234, Roll out one warm ACK research worker and prove cache-hit mission latency #243, Compile a multi-event causally available BTC 5m Polymarket evidence cohort #308, and Acquire a bounded multi-event BTC 5m market cohort in ACK #313 under this PRD.
• A three-event cohort proves mechanics; at least 24 hours is required for an operational research trial, and materially longer history is required before an alpha claim.
• Completion means a real immutable cloud readback, not CI, collector health, image build, or Job status alone.