Uh oh!
There was an error while loading. Please reload this page.
Prism v3: significance-gated emission, default-off - #168
Merged
Conversation
Adds PRISM_EMISSION_MODE=sig alongside wta/top3: - paired.rs: paired per-example displacement test. Absolute dead zone (0.01), win-rate bar 0.55 at a 99% bootstrap LCB, 10k resamples, fixed seed + in-crate SplitMix64 so leaves are reproducible (consensus). - frontier.rs: per-axis elite archive over g1..g8. Operator-owned cells, derived from stored measurements, so a miner cannot invent an axis. - sig.rs: champion 60% (floor 50%, difference burns), band 15/10/5%, 10% exploration pool, weight EMA, tail floor, burn remainder. Tenure decays the economic floor only; the statistical term never does. - rerun.rs: unannounced champion re-run schedule (keyed hash: unpredictable in advance, verifiable after) + regression verdict. Burn is real: BUNDLE_SPEC 6.4 normalizes a challenge's positive leaves, so the remainder is emitted as a uid-0 leaf that 6.5 drops and burns. Ships OFF. The bootstrap sees eval-item variance only, never sigma_seed, so its LCB is overconfident; a significance test on a wrong SE is worse than honest WTA. Must not be enabled before sigma_seed is measured. Default path proven unchanged by test (bit-identical to legacy WTA).
The mirror-gap contamination penalty is inert by construction in the public_dev tier: build_mirrors sets mirror = dict(public), so the gap is identically 0. That was labelled in a comment but nothing surfaced it, so a scored public_dev run looked contamination-checked when it was not. - rollup.mirror_report: emits battery.mirror_defence with contamination_checked / inert_pairs / live_pairs / reason, and warns on the console when inert. Inertness is measured per pair (both sides are the same measurement), not inferred from the tier, so a partially staged pack is visible instead of averaged away. - finalize: PRISM_EVAL_REQUIRE_PRIVATE=1 refuses to score a run with no contamination evidence (ContaminationUnchecked). Implied by PRISM_EMISSION_MODE=sig, since the sig rule grants a protected champion share on measured evidence. An absent flag counts as unchecked, so an older harness cannot pass by silence. - G2: PRISM_EVAL_G2_TASKS restricts scored tasks (default: all 8, so v0/v1/v2 anchor sets keep passing completeness), and per-task mean_gold_nll is emitted as observed, giving a live signal on axes whose accuracy is pinned at chance. Both knobs default off: CI, Sim, local-e2e and the live path are unchanged.
Research (non-normative, docs/spikes/**): - research/15-incentives-and-landscape.md — emission collapse, copy economics, anti-Goodhart. Front-matter marks it non-normative, points at PRISM.md for what the system does, and carries an errata: it was produced against /root/gbase, where top3 and the opt-in owner split do not exist. - research/16-budget-and-measurement.md — compute currency + the sigma_seed programme that is the prerequisite for enabling the sig mode. - research/incentives-sources/ — the three underlying evidence reviews, preserved with their primary-source URLs. - The scaling report was already filed as appendix 14; not duplicated. Normative: - PRISM.md gains a v3 significance-gated emission section: the rule, its knob, default-off status, why burn needs a uid-0 leaf, and the hard sequencing constraint (do not enable before sigma_seed is measured). - external-miner/prism.md gains two miner-facing sections: what Prism does and does not claim (it ranks at a pinned small budget; Tay et al. is direct evidence such rankings need not transfer to scale), and a plain explanation that beating the champion needs a measurement-uncertainty margin, not a better point estimate — plus that nothing pays for merely being different.
Reviews the three inherited v3 commits and closes what they left open. Ownership: reverts the edits to prism-eval-store (sibling-owned) and reimplements the contamination policy in prism-competition. The gate is NOT lost — sig-mode emission now fail-closes on its own, which is stronger than the finalize refusal it replaces. The one-line finalize insert is recorded for routing (contamination::FINALIZE_GATE_PATCH). evidence.rs — the missing bridge. paired.rs/frontier.rs defined the rule but nothing built it from stored data, so the mode was unreachable. prism_eval_metric persists per-cluster values for every scored run, so per-example incumbent data DOES exist and a real paired test is possible with no new state. Cluster ids are positional, so they only align when both sides scored the same slice: slice_id equality is required and a mismatch REFUSES rather than pairing positionally. No path falls back to aggregates. This is what makes the champion re-run load-bearing. Three defects fixed in sig.rs: - empty field set burn_bps=0 while claiming to burn everything, so conserves() was false and the burn leaf carried nothing. - the EMA resurrected hotkeys absent from this round's credits. Their share counted as allocated (not burned) but apply_significance emits no leaf for them (not paid), so the mass silently redistributed to the champion at BUNDLE_SPEC 6.4. Now only live credits are allocated. - contamination evidence was unenforced. An unchecked round now pays nobody and burns the full share: the protected 60% is granted on measured evidence and public_dev has none by construction. MIN_DECIDED 30 -> 100. The stated criterion is SE(win_rate) <= ~5%; at p=0.55, n=30 gives 9.1% and n=100 gives 4.97%. A test asserts the arithmetic so a future edit has to face it. prism-emit computes the plan once instead of twice, so allocation and burn cannot drift. Docs: same-slice limit, the two contamination levels and why the private tier is effectively mandatory under sig, corrected #DECIDED. Miner doc gets the 100-example floor and the same-slice consequence. G2 v3 anchor changes filed as a routing request (anchor JSON is sibling-owned). Gates: fmt, clippy, 1263 tests, deny, loc-cap, consensus-lint, spec-check, design-check, external-docs-check, all harness tests + BATTERY SMOKE OK. Default path unchanged: prism-eval-store is byte-identical to the base branch.
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
prism-v2.1-scoring. AddsPRISM_EMISSION_MODE=sigalongsidewta/top3, default-off, plus the per-axis elite archive, unannounced champion re-runs, and the loud/fail-closed mirror-defence fix.The rule
Champion 60 % (floor 50 % when the win is real but sub-premium; the difference burns) held until displaced by a paired per-example test on the same private slice: absolute dead zone 0.01, win rate ≥ 0.55 at a 99 % bootstrap LCB (10 000 resamples, fixed seed
20260816), mean gap ≥ 0.01, ≥ 100 decided examples. Ranks 2–4 take 15/10/5 %; 10 % splits across ≤5 gate-passing entries holding any per-axis frontier; the remainder burns. Plus a weight EMA (α = 0.5) and a 100 bps tail floor. Tenure decays the economic floor only — the statistical bar is a truth condition and never decays.Margins are absolute, never relative: D nats is D nats of evidence whether the loss is 0.02 or 2.0, and a relative margin collapses exactly where the metric saturates. The win-rate bar is 0.55 and deliberately not higher — a genuinely better architecture with wide per-example spread sits near 0.55, so a higher bar selects for low-variance submissions rather than good ones.
Motivation is arithmetic, not fairness: under WTA a functional clone has identical true quality, so it takes the whole share with p ≈ 0.5 — EV ≈ 50 % of emissions for one pod — which makes an evadable copy detector load-bearing. Significance gating cuts that to < 5 % mechanically. It protects the champion share, not the band, so the copy gate stays necessary.
Why it ships off, and the constraint on turning it on
The bootstrap measures eval-item variance only. Training-seed variance (
σ_seed) is absent because each submission is trained exactly once, and a seed change alone re-ranks NAS architectures at Kendall τ = 0.48. The lower bound is overconfident by construction, and a significance test on a provably wrong standard error is worse than honest WTA — it lends false statistical authority to a biased ranking.sigmust not be enabled beforeσ_seedis measured and published. Enabling it is a governance action; the prerequisite order is indocs/PRISM.md.The default path is unchanged and a test enforces it: with the env unset,
build_epoch_leavesreproducesapply_wta(competition_scores(..))leaf-for-leaf, andprism-eval-storeis byte-identical to the base branch.Per-example data for a real paired test
prism_eval_metricpersists eachorg.*metric with its per-cluster values for every scored run, and the harness records one cluster id per item. So per-example data does exist for both sides, including past champions — no new state.evidence.rsbuildsSigContext/PairedInput/EliteArchivefrom those rows.One real limit, enforced rather than documented-and-ignored: cluster ids are positional, so they align only when both sides scored the same asset slice. A rotating slice makes ids collide numerically while referring to different items.
slice_idequality is required and a mismatch refuses (PairedRefusal::SliceMismatch) — there is deliberately no fallback that pairs aggregates, since comparing independently-bootstrapped levels is what the paired design exists to avoid. This is what makes the champion re-run load-bearing rather than optional.Mirror defence: loud, and fail-closed where it matters
The contamination penalty is inert by construction in
public_dev(build_mirrorsmakes the run its own mirror, gap ≡ 0). It was honestly labelled in a comment but nothing surfaced it, so a scoredpublic_devrun looked contamination-checked. Now the harness emitsbattery.mirror_defence(contamination_checked,inert_pairs,live_pairs, reason) and warns on the console, andsigmode pays nobody on an unchecked round — full burn. Absent flag counts as unchecked, so an older harness cannot pass by silence.The default is off because
public_devis the tier CI, Sim and local-e2e all run in, and every one of those runs is supposed to have an inert mirror — a default-on gate would fail-closed on the whole test matrix rather than on a real risk. Strictest posture that keeps CI green: loud always, refused where a protected share is at stake.Not scored: novelty
The exploration pool pays per-axis frontiers (
g1..g8), not measured difference. Nobody has made "pay for being different" work — Numerai marketed it and implemented marginal contribution, and the component that got exploited was the rank-shaped bonus. A novelty distance is faked by renaming variables; being best at associative recall is not. Cells are operator-owned, so a miner cannot invent a ninth axis.OWNER_ARCH_CREDIT_ENABLEDstaysfalse(tea.xyz was farmed to >150 000 spam packages).Tested
Default-path bit-identity; determinism and integer conservation across field sizes; dead-zone, win-rate and mean-gap logic; champion-hold vs displacement; clone-cannot-take-the-crown; unchecked-round full burn; every allocated share reaching a leaf; cross-slice refusal.
Gates:
fmt,clippy -D warnings, 1263 tests,cargo deny,loc-cap,consensus-lint,spec-check,design-check,external-docs-check, all harness python tests,BATTERY SMOKE OK.Needs routing (owned elsewhere)
docs/spikes/prism-v3/g2-v3-routing-request.md— the exact G2 task/weight changes foranchors/v3.json(four of eight sub-metrics normalize to a constant 0 for the whole field, carrying half of G2's weight while measuring nothing), and the optional finalize-time contamination gate. Harness-side support for both is implemented here.