fix(research): close sealed-holdout diagnostic bypasses - #1159
fix(research): close sealed-holdout diagnostic bypasses#1159proerror77 wants to merge 4 commits into
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1fb49cc867
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| _ => None, | ||
| }; | ||
| let finalization = None; |
There was a problem hiding this comment.
当 GP v1–v3 公式轮次产生通过 replay gate 的候选时，这里无条件清空 finalization，删除了其唯一的封存 holdout 终评路径；但新的独立终评仍在 mission_campaign/final_evaluation.rs:430-483 仅接收 supervised_replay_gate_passed == Some(true) 的监督模型，公式轮次会被跳过并以“no supervised replay-qualified round winners”失败。因此 README 所述“公式 lane 通过 --final-evaluation 终评”实际不可达，已保留的公式研究无法产出最终证据或 promotion；应在移除旧入口的同时，将公式 winner 的冻结与终评迁移到签名授权的独立终评流程中。
AGENTS.md reference: AGENTS.md:L12-L15
Useful? React with 👍 / 👎.
| }; | ||
| store.put_registry_revision(&revision)?; | ||
| Ok(revision) | ||
| bail!("public evaluate cannot read sealed holdout without the global create-once claim") |
There was a problem hiding this comment.
当旧版或手工导入的 Mission 已有 canonical walk-forward winner、但尚无 mission_evaluation_protocol revision 且没有 sealed revision 时，执行会先在第 418 行调用 bind_mission_evaluation_protocol 永久写入 registry，然后才在这里返回“没有 global claim”的错误。这样一个声称失败且只读的 evaluate 调用仍会改变 append-only 治理状态，后续调用会观察到并受该绑定约束；应在确认可返回已有 sealed evidence 后仅校验绑定，或在拒绝路径之前避免创建 revision。
AGENTS.md reference: AGENTS.md:L92-L94
Useful? React with 👍 / 👎.
| /// Stop before opening sealed holdout, including for the formula lane. | ||
| /// Required unless `--final-evaluation` supplies an independent grant. | ||
| #[arg(long, required_unless_present = "final_evaluation")] |
There was a problem hiding this comment.
当操作者继续使用仓库提供的 deployment/aliyun/research/k8s/alpha-mission-job.example.yaml:59-71 启动普通 Campaign Job 时，该模板没有传入现在必需的 --pre-holdout，因此 Clap 会在进入 worker 前直接拒绝命令；本提交自己的 cli.rs:1457-1460 测试也明确断言无该参数时解析失败。已检查生成式 dispatch：它会添加此参数，受影响的是仍被文档描述为 tokenless campaign-execute shape 的手工示例；应同时更新该模板，否则复制部署示例得到的 canonical Campaign Job 无法启动。
AGENTS.md reference: AGENTS.md:L12-L15
Useful? React with 👍 / 👎.
Require --pre-holdout for campaign-execute unless --final-evaluation, keep Direct mission execute search-only, and refuse public evaluate from reading sealed holdout without the global create-once claim. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
Admit formula replay winners on signed independent --final-evaluation, stop public evaluate from writing the protocol registry, and require --pre-holdout on the hand-started Campaign Job example. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
Elide a needless lifetime, allow bounded argument counts on the final-evaluation holdout helpers, and lock formula frozen-selection identity plus unreserved walk-forward fallback. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
a27408c to
086c4c1
Compare
Parse existing sealed evidence before requiring a bound protocol so read-only evaluate still reports legacy records as malformed, and rustfmt the frozen-selection identity helper. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
Change
Sealed holdout is no longer openable from research diagnostic or default Campaign surfaces.
mission campaign-executewithout--pre-holdoutfails closed. Formula-lane execute no longer auto-finalizes. Holdout still opens only through signed independent--final-evaluation, which now admits both supervised and formula replay winners.mission execute(Direct) is search-only: it can run GP/subset/replay diagnostics but does not claim holdout or write promotion. Direct still rejects feature fields with no live formula semantics.evaluateis read-only: it returns existing sealed registry evidence and otherwise refuses without the global create-once claim. Failed evaluate does not bindmission-evaluation-protocol.--pre-holdout.This is one holdout-gate behavior, so it stays in a single PR.
Issue relationship
None
Validation
git diff --checkcargo test -p alpha-harness --locked -- winner_lane_admits_formulacargo test -p alpha-harness --locked -- campaign_execute_without_pre_holdout_refusescargo test -p alpha-harness --locked -- direct_execute_does_not_opencargo test -p alpha-harness --locked -- public_evaluate_cannot_readcargo test -p alpha-harness --locked -- execute_rejects_features_that_have_no_live_formula_semanticscargo test -p alpha-engine --locked -- independent_selection_falls_back_to_walk_forwardcargo test -p alpha-domain --locked -- frozen_selection_ids_accept_formulacargo clippy -p alpha-harness --locked --tests -- -D warningscargo clippy -p alpha-engine -p alpha-domain --locked --tests -- -D warningsRuntime impact and rollback
None. Research CLI/gates only; no live, collector, or runtime mutation. Rollback is revert of this commit. Automatic research-image publication may follow merge; that is not deployment authority.