docs(architecture): define ready research contract - #327
Conversation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe prediction markets architecture document formalizes a verifier-driven, continuously-ready research contract and clarifies identity, evidence, authority, rejection, and completeness rules. The governed baseline is limited to verified BTC five-minute episodes, with other horizons and SOL requiring separate contracts. ChangesPrediction markets research contract
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related issues
Possibly related PRs
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5847d988fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | Cohort manifest | Ordered ready-catalog entry digests, event-disjoint partition assignment, common-time boundary, label-availability cutoff, and causal projection rules. | | ||
| | `ResearchSnapshot` | Cohort-manifest digest plus the digests of the exact evaluator-visible bytes. It is materialized and cached before Mission admission. | | ||
| | Mission | Mission SHA over product family, typed task, run mode, authority profile, cohort partition, snapshot digest, task horizon where applicable, and all evaluator/search parameters. Raw collector paths are forbidden. | | ||
| | Runtime image | Exact repository revision, release-binary digest, and immutable OCI image digest selected by the admitted Mission. A mutable tag is not an identity. | | ||
| | Result receipt | Mission SHA, snapshot digest, image digest, lifecycle timestamps, and content digest of a create-once result bundle independently rehashed after publication. | |
There was a problem hiding this comment.
This single issue-level decision record combines cohort/snapshot construction, Mission/evaluator execution, runtime-image admission, and result publication into one binding contract. These stages have independent acceptance and rollback boundaries, and the repository specifically requires them to remain separate Polymarket issues; closing #315 with all of them bundled allows one approval to be treated as authority for unrelated downstream work. Split the stage-specific contracts into separate issues and leave only their shared boundary or cross-references here.
AGENTS.md reference: AGENTS.md:L39-L42
Useful? React with 👍 / 👎.
| | Mission | Mission SHA over product family, typed task, run mode, authority profile, cohort partition, snapshot digest, task horizon where applicable, and all evaluator/search parameters. Raw collector paths are forbidden. | | ||
| | Runtime image | Exact repository revision, release-binary digest, and immutable OCI image digest selected by the admitted Mission. A mutable tag is not an identity. | |
There was a problem hiding this comment.
The Mission SHA field list omits the repository revision, release-binary digest, and OCI image digest even though the next row says that the admitted Mission selects that exact runtime image. Consequently, the same admitted Mission identity can be executed or retried with different code while downstream records still treat it as the same Mission; recording the image only in the result receipt is too late to make admission or execution reproducible. Include the immutable image identity in the Mission SHA, or introduce an admitted execution identity that binds the Mission and image before creating the Job.
Useful? React with 👍 / 👎.
| | Typed task | Allowed label | Required evidence | | ||
| | --- | --- | --- | | ||
| | `settlement_probability` | One official binary episode outcome: `UP` or `DOWN`. | Exact market contract, causal Chainlink opening/expiry observations, official resolved outcome and availability clock, plus event-disjoint Brier score, log loss, calibration, and settlement-PnL evidence. Token markouts and fills are not settlement labels. | | ||
| | `up_execution` | UP-token fill outcome, realized execution price/slippage, or executable markout at an explicit 5, 10, 15, or 30 second horizon. | Only the bound UP token's point-in-time book/trades, request and sequence evidence, fees, latency, and declared fill/queue assumptions. Official settlement is not an execution label. | |
There was a problem hiding this comment.
When an execution task chooses realized fill price or slippage rather than a simulated markout, the listed evidence contains only public books/trades and declared fill or queue assumptions; it does not require an immutable order, intent, queue position, or fill record from which a realized label can be verified. Such an episode can therefore be classified ready and evaluated against an assumed value presented as realized execution. Require order/fill provenance tied to the decision for realized targets, or explicitly define these targets as simulated and bind the simulation model and assumptions into the task identity.
Useful? React with 👍 / 👎.
| - One named runtime controller may submit, replace, delete, deploy, or otherwise | ||
| mutate catalog and cloud runtime resources. All other agents and operators are | ||
| read-only until an explicit handoff. |
There was a problem hiding this comment.
Granting the runtime controller authority to replace, delete, or otherwise mutate catalog resources conflicts with the earlier guarantees that only the independent verifier may append classifications and that rejected or partial receipts remain immutable. If this authority includes catalog entries, a controller can remove or replace admission evidence without a new verification receipt, invalidating cohort digest availability and the audit trail. Restrict the controller to catalog-service infrastructure while explicitly reserving classification-entry writes to the verifier.
Useful? React with 👍 / 👎.
| | Verifier receipt / ready catalog entry | Producer manifest digest; verifier binary, configuration, and policy digests; classification and reason codes; exact product, episode, UP token, DOWN token, supported-task, and coverage identities. | | ||
| | Cohort manifest | Ordered ready-catalog entry digests, event-disjoint partition assignment, common-time boundary, label-availability cutoff, and causal projection rules. | |
There was a problem hiding this comment.
A catalog entry records the verifier-policy digest, but cohort construction only pins entry digests and does not require their policies to be currently accepted. Because classifications are immutable, an episode marked ready by an older policy remains selectable even after a policy update closes a completeness or provenance gap, allowing a new Mission to pass admission with stale readiness semantics. Require the cohort or Mission admission policy to pin and enforce an allowed verifier-policy set, or require a new verification receipt under the selected policy.
Useful? React with 👍 / 👎.
| | Run mode | Authority | | ||
| | --- | --- | | ||
| | `pipeline_smoke` | May prove producer-to-result schema compatibility, admission, cache use, evaluator start, publication, and digest readback with a minimal complete cohort. It cannot emit an alpha, generalization, promotion, or profitability verdict. | | ||
| | `research_trial` | May run the typed evaluator and shared MCTS kernel only on event-disjoint train/validation/held-out cohorts. Held-out labels, metrics, and feedback cannot alter candidate search, fitting, selection, or stopping. It may publish research evidence, not activation authority. | |
There was a problem hiding this comment.
The trial now has separate train, validation, and held-out cohorts, but the contract pins only one common-time boundary and the subsequent boundary definition separates only training from validation. Event-disjointness alone still permits a validation event used for search or stopping to occur after an earlier held-out event, leaking future regime information while satisfying the stated checks. Bind an ordered validation-to-held-out boundary as well, or explicitly define and validate the complete temporal ordering of all three partitions.
Useful? React with 👍 / 👎.
Change contract
Record the binding BTC x 5m continuously-ready research identity, admission, task-authority, ownership, and counterexample contract in the shared prediction-market decision record.
Out of scope
Collectors, catalog storage, snapshot or Mission implementation, research logic, CI, deployment, cloud/runtime mutation, data backfill, old-artifact compatibility, Paper, Shadow, Live, promotion, and profitability.
Dependency or merge order
None.
Focused validation
git diff --check origin/main...HEADnpx --yes markdownlint-cli@0.45.0 docs/architecture/PREDICTION_MARKETS.md --disable MD013(MD013 disabled for the existing table-heavy document convention)Rollout/rollback impact
No runtime rollout or authority change. Roll back by reverting this documentation commit; no artifact or data migration is required.
Closes #315
Summary by CodeRabbit