feat(research): expose native supervised model comparison metrics - #1152
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 15ffa16073
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Bring in #1152 so GitHub will accept squash-merge; no prediction-markets overlap. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
Change
Supervised CEX rounds now emit a content-bound JSON/CSV comparison of every evaluated model. The report preserves native IC/Rank IC, ICIR/Rank ICIR, unannualized Sharpe, return, drawdown and turnover, and adds prediction amplitude, position amounts and explicitly qualified calendar-day/benchmark metrics. Incomplete days, missing benchmarks and degenerate ratios remain unavailable rather than becoming zero or annualized estimates.
The read-only
mission model-metricscommand reuses the same implementation for completed artifacts. It does not train or authenticate settlement. Campaign readback requires both reports when supervised selection exists. It verifies original feature bytes against the SHA256 inside the admitted Mission, reconstructs the pre-holdout data view and partition, and recomputes positions and ledger accounting without fitting models. Rehashing a forged report, altered ledger and ZIP cannot replace the admitted prices, timestamps and costs. Source cost/equity/turnover inconsistencies are rejected, and report outputs only reuse identical bytes.Issue relationship
Closes #1151
Validation
Red: the existing profitable supervised Campaign regression failed because the unified report was absent.
Green: the same end-to-end regression passes, including JSON/CSV output, native metric preservation, inconsistent accounting, removed sidecars, forged ICIR, shifted timestamps, balanced return redistribution and tampered admitted feature bytes; every tampering case rehashes the outer bundle.
cargo test -p alpha-engine --locked model_metrics: 6 passed;cargo test -p alpha-harness --locked metrics: 5 passed.Scoped Clippy for alpha-engine and alpha-harness, all targets, with warnings denied; diff whitespace check passed.
Built and reran the CLI at
84329b6bb1179a3437fa398405a79ca110d9b91fon 54 retained real backtests plus 18 original selections. It reproduced identical immutable JSON/CSV outputs: 18 cohorts / 54 rows, exact native metric equality, and explicitly unavailable daily statistics for partial-day evidence.An additional read-only check exercised the new production verifier against all 18 retained real bundles and 54 model ledgers. Original admitted feature hashes and pre-holdout dataset hashes matched; timestamps, positions and accounting were recomputed successfully without model fitting. Temporary test instrumentation was removed and the committed source restored byte for byte. Original study outputs were unchanged; no new training, research Job, or holdout evaluation occurred.
Integrated main at
01d07a0d3788fc9672220770603da47cba04dc6dwithout conflicts. The alpha-harness source is byte-identical to the locally verified84329b6implementation; the only integration changes are the independently landed prediction-market snapshot fix.All three required GitHub checks passed on integration head
e48d59cfb385f21f4b648d886350a6636c8e7986: Monorepo CI gate, Prediction Markets CI gate, and Security Summary Report. Both review threads are resolved.Runtime impact and rollback
Research output and readback/reporting behavior only. Model selection, trial counts, fee/risk gates and deployment authority are unchanged. LightGBM/XGBoost/TCN fitters are subsequent work, not implemented here. Merge may trigger configured artifact publication; it is not a runtime cutover. Revert this commit to remove the new producer/reporting behavior; original candidate and backtest evidence remains authoritative and preserved.