feat(research): switch prediction runner to shared MCTS - #201
Conversation
|
Warning Review limit reached
Next review available in: 39 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
413bf79 to
2800d96
Compare
06e1b16 to
896efc7
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 06e1b162db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| &mut evaluator, | ||
| ) | ||
| } else { | ||
| run_or_resume_prediction_mcts( |
There was a problem hiding this comment.
When the default command is pointed at a resumed legacy result bundle—or any non-empty output directory without prediction-mcts-state.json—this path initializes a fresh MCTS run alongside the existing files. Unlike run_or_resume, run_or_resume_prediction_mcts does not call reject_unbound_output_directory, so a paused legacy run resumed without the new --legacy-loop flag silently starts over and can upload a bundle containing evidence from two controllers or missions instead of failing closed.
Useful? React with 👍 / 👎.
| &mut evaluator, | ||
| ) | ||
| } else { | ||
| run_or_resume_prediction_mcts( |
There was a problem hiding this comment.
For every non-baseline mission, switching the default to this runner drops the governed LLM audit trail: run_or_resume_prediction_mcts parses ProposalCallOutput.raw_response into blends and persists only those blends plus an optional failure string, discarding the exact response, provider, model, and usage. The legacy default records those fields as durable append-only response evidence, so default MCTS runs can no longer audit or reproduce the advisor call from their result bundle.
Useful? React with 👍 / 👎.
| training_candidate_json: None, | ||
| selected_candidate_json: Some(candidate_path), | ||
| }, | ||
| timeout, |
There was a problem hiding this comment.
When the final training evaluation consumes the remaining wall-clock budget, or a baseline/held-out pause is resumed after its persisted deadline, the MCTS controller still calls this adapter with a zero timeout; PredictionEvaluator::evaluate then spawns the evaluator and immediately kills it, records another pause, and repeats that behavior on every resume. The legacy loop checks its budget before entering each frontier, so the new default needs the same gate before baseline, advisor, and selected evaluation rather than attempting work after expiry.
Useful? React with 👍 / 👎.
| eprintln!("ERROR: load governed research snapshot: {error:#}"); | ||
| std::process::exit(2); | ||
| }); | ||
| validate_prediction_run_inputs(&mission, &snapshot).unwrap_or_else(|reason| { |
There was a problem hiding this comment.
For a checkpoint created before this change, the saved mission pins the previous policy ID (the checked-in missions used sha256:7fe89...), but this unconditional validation now compares it with the new MCTS-inclusive policy ID (sha256:f475...) before consulting legacy_loop. Consequently, even an operator who supplies --legacy-loop cannot resume an existing paused legacy checkpoint—the command exits with a policy mismatch—so the advertised rollback path only starts legacy logic for newly rewritten missions rather than resuming the runs it is meant to preserve.
Useful? React with 👍 / 👎.
Closes #187
Change contract
Make
monday-prediction-researchuse the durable shared-kernel prediction controller, while preserving--legacy-loopas an explicit rollback path.Out of scope
Deleting legacy state, changing settlement metrics, Alpha Search artifact format, collector/snapshot, result publication, promotion, execution, risk, or deployment.
Dependency / merge order
Stacked on #200, which is stacked on #199. Merge #199, then #200, then this PR.
Focused validation
cargo test -p ploy-research prediction_mcts --no-default-featurescargo test -p ploy-research --bin monday-prediction-research --no-default-featurescargo test -p ploy-research --bin monday-prediction-evaluator --features db --no-default-featurescargo test -p ploy-research checked_in_btc_and_sol_templates_pin_current_brief_and_rust_policy --no-default-featurescargo clippy -p ploy-research --no-default-features --no-deps -- -D warningscargo fmt --checkRollout / rollback impact
Default CLI uses the new checkpoint;
--legacy-loopdirectly resumes the old controller and its own checkpoint without translation. Reverting restores the old default.Review
Matt standards and spec reviews passed after the split. The exact selected candidate is persisted, forwarded, and validated against the governed prior before held-out evaluation.