feat(research): evaluate authenticated MCTS tasks

[proerror77]

Change contract

Evaluate the selected canonical shared-MCTS candidate against authenticated settlement, Up-execution, or Down-execution snapshot views; emit a typed, self-digested evaluator artifact and fail closed when any admitted decision, joint binary book, task identity, candidate identity, requested horizon evidence, snapshot-bound execution setting, or settlement metric is missing, mismatched, or non-finite.

This is the already-specified evaluator layer of #324; a separate PRD is unnecessary because #324 and its OWNER clarification define the behavior contract and authority boundary.

Out of scope

• Typed result receipts and the parent experiment manifest, which remain the next focused Run settlement, Up, and Down Mission v3 tasks through the shared MCTS kernel #324 layer.
• The Prove the producer-to-result contract with one three-event CI fixture #325 three-event CI fixture, Run the exact-main Polymarket baseline evaluator and MCTS in cloud #233 ACK trial, and Publish the Polymarket baseline E2E completion proof #234 immutable publication.
• Collector, runtime, deployment, Paper/Shadow/Live, CEX/CEX: add an MCTS research adapter for time-series HFT alpha discovery #205, alpha-harness, and hft-search-kernel changes.
• Profitability, promotion, or candidate-optimization claims; reduced authority has exactly one canonical candidate from merged fix(research): collapse reduced MCTS to canonical baseline #407.

Dependency / merge order

#405 and #407 are merged. This PR is rebased directly onto main at 5f7ca6c; no stacked dependency remains.

Atomic scope exception

The complete diff is 1007 additions and 13 deletions across one substantive evaluator file plus two required policy-digest template syncs. Codex reviewers /root/review_324_joint_spec and /root/review_324_joint_standards explicitly approved the atomic exception. Snapshot-bound settings, exact decision and horizon coverage, joint-depth validity, finite typed metrics, artifact self-validation, and post-compute deadlines are one fail-closed evaluator trust boundary. Splitting would permit an evaluator with known incomplete authenticated guarantees to merge or roll back independently.

Focused validation

• cargo fmt --package ploy-research -- --check
• cargo clippy --locked -p ploy-research --features ml --all-targets --no-deps -- -D warnings
• cargo test --locked -p ploy-research: 347 library tests plus 4 binary tests
• Current policy ID readback equals both BTC/SOL templates: sha256:02c3a3b1567b204b2be922e94bf4afa6500fdb83fbe53df13b6df612049a1a13
• git diff --check
• Parallel spec and standards code reviews: CLEAN

Targeted counterexamples cover snapshot-bound stake and quote age, missing requested-horizon observation or book, hashed but invalid joint depth, empty settlement held-out evidence, post-compute deadline expiry, exact-boundary finite log loss for either outcome, missing settlement joint quotes, missing sibling execution books, task metric substitution, candidate or task identity drift, label isolation, and artifact digest tampering.

Rollout / rollback impact

Research-only artifact behavior. No collector, runtime, execution, deployment, or live-trading state changes. Roll back by reverting this PR; existing authenticated snapshots and shared-kernel state remain immutable and fail closed.

Refs #324

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR updates BTC and SOL mission snapshot identifiers and substantially revises authenticated Prediction MCTS evaluation with typed settlement/execution metrics, schema-versioned artifact validation, deadline checks, persistence validation, and expanded tests.

Changes

Authenticated Prediction MCTS evaluation

Layer / File(s)	Summary
Typed artifact and metrics contracts rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs	Authenticated artifacts now include schema, candidate digest, typed metrics, and canonical digest validation; snapshots expose quote-age and stake settings.
Settlement and execution metric computation rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs	Task-specific computation validates authenticated quotes, decisions, horizon evidence, order-book depth, and coverage before calculating settlement or side-specific execution statistics.
Evaluator integration and validation rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs, rust_hft/prediction-markets/config/research_missions/polymarket-*-5m.example.json	Evaluator paths enforce deadlines and identity checks, emit typed evidence, validate persisted artifacts, expand metric tests, and update BTC/SOL policy snapshot identifiers.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Mission
  participant Evaluator
  participant Metrics
  participant Artifact
  Mission->>Evaluator: provide task and candidate
  Evaluator->>Metrics: compute settlement or execution metrics
  Metrics-->>Evaluator: return typed metrics
  Evaluator->>Artifact: construct and validate held-out artifact
  Artifact-->>Evaluator: persist validated artifact

Loading

Possibly related PRs

• proerror77/monday#199: Extends the related training settlement evidence and prediction-report flow.
• proerror77/monday#384: Introduces task-typed training evidence and identity validation used by this evaluation flow.
• proerror77/monday#387: Modifies the authenticated Prediction MCTS flow and cohort gating used by this implementation.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: authenticated MCTS task evaluation.
Description check	✅ Passed	The description covers all required template sections and includes validation, merge order, rollout, and scope exception details.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/mission-v3-mcts-324-evaluator

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af2c53b3c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs (2)

265-283: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Hardcoded stakes_usd: vec![15.0] / min_bucket_observations: 1 override the module defaults without an explanatory comment. Consider extracting a named constant documenting why $15 / per-decision granularity is the right choice for authenticated evaluation (vs. the bucketed defaults of [1.0, 3.0, 5.0, 10.0, 15.0] / 20).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs`
around lines 265 - 283, The execution_task_metrics options use unexplained
authenticated-evaluation overrides. Extract the $15 stake and per-decision
min-bucket settings into clearly named constants near the module defaults, add a
concise comment documenting why authenticated evaluation requires them instead
of the bucketed defaults, and use those constants in
FullDepthExecutionMatrixOptions.

---

854-1139: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Solid coverage for label-isolation and missing-quote/book fail-closed behavior. Consider adding one more counterexample for an out-of-range prediction_horizon_secs (e.g. 21) asserting builtin_task_metrics/execution_task_metrics returns Err rather than zeroed metrics, tying into the fail-closed gap flagged on Line 172-200/265-405. Happy to draft this test if useful.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs`
around lines 854 - 1139, The existing tests lack coverage for invalid execution
horizons. Extend the authenticated metrics tests around `builtin_task_metrics`
and the `mission` helper with an UpExecution mission using
`prediction_horizon_secs` set to 21, and assert the metrics call returns `Err`
rather than zeroed metrics; ensure the underlying `execution_task_metrics` path
remains fail-closed.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs`:
- Around line 202-263: Update settlement_task_metrics to ensure the
mean_log_loss calculation remains finite when q is exactly 0.0 or 1.0. Adjust
the log-loss computation around the existing q.ln() and (1.0 - q).ln() calls by
clamping probabilities or handling boundary outcomes explicitly, while
preserving the current binary-label scoring behavior and ensuring the resulting
artifact can be serialized and hashed consistently.

---

Nitpick comments:
In
`@rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs`:
- Around line 265-283: The execution_task_metrics options use unexplained
authenticated-evaluation overrides. Extract the $15 stake and per-decision
min-bucket settings into clearly named constants near the module defaults, add a
concise comment documenting why authenticated evaluation requires them instead
of the bucketed defaults, and use those constants in
FullDepthExecutionMatrixOptions.
- Around line 854-1139: The existing tests lack coverage for invalid execution
horizons. Extend the authenticated metrics tests around `builtin_task_metrics`
and the `mission` helper with an UpExecution mission using
`prediction_horizon_secs` set to 21, and assert the metrics call returns `Err`
rather than zeroed metrics; ensure the underlying `execution_task_metrics` path
remains fail-closed.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 043dfc0e-803c-42e2-b910-f0877dcc65d4

📥 Commits

Reviewing files that changed from the base of the PR and between 5f7ca6c and 5c081ca.

📒 Files selected for processing (3)

• rust_hft/prediction-markets/config/research_missions/polymarket-btc-5m.example.json
• rust_hft/prediction-markets/config/research_missions/polymarket-sol-5m.example.json
• rust_hft/prediction-markets/crates/ploy-research/src/prediction_mcts_authenticated.rs