Uh oh!
There was an error while loading. Please reload this page.
Prism: attested-FLOPs budget currency, dual caps, anchors v3 - #167
Merged
Conversation
The budget a submission spends becomes ATTESTED FLOPS, measured by the harness. Wall-clock survives only as an anti-DoS bound. The miner reports no number anywhere in this path. Why not wall-clock as the currency: a fixed wall makes MFU a *scored* quantity, so two identical architectures differ in score by kernel maturity (no sm_120 FlashAttention cubins, Triton version rent), and a looped model at r=4 pays ~3.3x FLOPs/token and therefore sees ~3.3x fewer tokens -- charged to the architecture as if it were a quality defect. FLOPs measured from the REALIZED dispatch graph price looping, MoE sparsity and vocabulary size automatically, so the budget adapts to the architecture class with no tier to declare and none to shop for. prismlib/flops.py (new): - probe_flops_per_token: median over FLOPS_PROBE_SAMPLES=8 harness-driven fwd+bwd passes under torch.utils.flop_counter.FlopCounterMode, on batches drawn from the real train stream at SECRET indices. Above FLOPS_PROBE_CV_MAX=0.15 the estimator switches from median to MAX, so input-dependent cost (a MoE routing cheaply on probe-shaped inputs, an early exit) is charged at its expensive branch instead of its cheap one. - probe_secret: the eval secret is staged only AFTER train, so the production path draws a fresh urandom secret in the train child. That is unpredictable-in-advance, NOT hidden from in-process inspection -- documented as such rather than overstated. - analytic_flops_per_token: the cross-check for the one hole counting cannot close. FlopCounterMode sees only what the dispatcher sees, and recipe-v10 lets miners install their own dependencies, so a fused Triton/CUDA op registered as one opaque dispatch is INVISIBLE. Analytic model is F_tok = 6*N_body*r_eff*active + 6*d*V + 12*L*d*S; a gap over FLOPS_ANALYTIC_GAP_MAX=0.25 sets flops_analytic_mismatch. Evidence for review, never a silent pass. - C = 6ND is wrong at this scale: at d=512,V=32768 the lm_head alone is ~36% of FLOPs/token, so 6*N_body captures only ~55% of the cost and 6ND overstates the affordable token count by 1.3-1.8x. Body params exclude embeddings AND the head (the head is charged once, by 6*d*V -- folding it into N_body and adding the term again was a ~2x error at small d). Only the body loops, which is why r=4 costs ~3.3x not 4x. - the quadratic attention term is charged ONLY when attention is detected, so a delta-net/SSM is not billed for a phantom cost it never pays -- that would manufacture a mismatch on exactly the architectures Prism exists to evaluate. - physically_possible + mfu: attestation above peak*n_gpu*wall*1.05 is internally inconsistent; n_gpu is attested, not declared, so a 1-GPU run cannot silently claim a 4-GPU allowance. prismlib/stream.py -- enforcement moves to where the tokens are handed out, not where the miner is asked to be polite. next_batch refuses to yield once a cap is reached and raises BudgetExhausted carrying WHICH cap bound. peek_batch(i) serves probe batches without advancing tokens_seen, bytes_seen or flops_spent: measuring the budget must not spend it. Also counts bytes_seen, because per-token CE is not tokenizer-neutral -- a tokenizer that compresses harder lowers CE/token without predicting better -- so a bits/byte curve needs the bytes behind the tokens. prismlib/train_v3.py -- probes before the first batch so the budget is armed from step one; catches BudgetExhausted and routes it to the SAME graceful checkpoint-then-eval path as finish_evaluation(). Reaching your own budget is the expected outcome; the predecessor (_CapExceeded) was caught by nothing and failed the whole run, which made spending the full budget a way to score zero. A probe that cannot run leaves the FLOPs cap DISARMED (the wall still contains the run) and records flops_probe_error -- a failed measurement is never read as a free model. Seeds from ctx rather than the constant, so the operator seed-variance sweep is not silently defeated by a child that re-seeds from the lattice. main.py -- _with_diag merges the train child's numeric org.diag.* into battery.metrics, because the attestation is measured in the TRAIN child while the composite reads org.* from the battery the EVAL child writes; without it the entire Phase-0 deliverable was emitted and dropped. Two constraints are load-bearing and asserted in tests: (1) only FINITE NUMERIC values are merged -- the Rust reader cannot parse a string, and org.diag.binding_cap is one, so non-numeric diagnostics travel in the top-level `budget` block instead; (2) it never CREATES a battery, because a diag-only battery would make the scorer run the composite and fail every declared group as missing -- Ineligible, lattice 0, instead of the correct "composite skipped". probes.py records bytes_seen/bytes_per_token/probe_bits_per_byte and flops_spent per curve point, so a byte-denominated G6 key names a quantity that exists. manifest.py attests the caps in force (it recorded neither the param cap nor any eval budget before, so two runs under different caps were indistinguishable after the fact). RECIPE_VERSION stays 2.0.0: this moves recipe_pin_hex() (harness bytes are pinned) but the semver bump is a release action with miner-facing docs, the public prism repo and the external-docs-check pin attached. Same precedent as the MAX_PARAMS raise. tests/test_flops_attestation.py (new, 23 cases, CPU-only): probe determinism for a fixed secret, the max-not-median switch under high CV, the opaque-body cross-check catching an under-counting kernel, no phantom attention term, loop/MoE scaling, dual-cap binding with the bound cap recorded, spend_fraction feeding the underspend guard, peek_batch not spending, byte accounting, and the metrics-plumbing constraints above.
Rust side of the attested-FLOPs currency: the constants, the gates, and one
coherent set of budget numbers across every layer that had drifted apart.
prism-budget (new crate): BindingCap, GateThresholds, BudgetFacts,
GateFailure, GateReport and check(). Split out of prism-pipeline::composite
because that module was within a few lines of the 1500 non-test LOC cap and
the budget gates are a self-contained concern with no dependency on
scoring; prism-pipeline re-exports every type, so the public API and the
serialized JSON shape are unchanged (precedent: prism-competition out of
prism-registry). prism-recipe::anchors re-exports GateThresholds too, so
the anchor registry and the scorer parse ONE schema rather than two structs
kept in sync by hand -- max_flops has to mean the same thing on both sides.
Two new gates:
- FlopsOverBudget: the currency.
- SpendBelowFloor at MIN_SPEND_FRACTION=0.5: the underspend guard. Without
it, an architecture that saturates early profits by stopping early,
because the compute-optimal frontier's slope is only ~-0.05..-0.10 nats
per e-fold -- buying less compute costs less score than it saves. Below
the floor the run is INELIGIBLE, not merely scaled, and no truncation
correction can ever be positive.
Both are skipped when the anchor set declares no max_flops (v0/v1/v2) or
the run carries no attestation. That is emit-then-declare, not laxity: a
gate declared before the harness emits its input would make every
in-flight submission ineligible. A wall-bound run that still cleared the
floor stays ELIGIBLE -- the wall is a safety bound, not a disqualification,
which is the whole reason binding_cap is recorded instead.
BUDGET RECONCILIATION -- three layers disagreed and now one set of numbers
is derived rather than duplicated:
- TRAIN_HOURS_CAP 6.0 -> 5.0 h, now the anti-DoS bound. Paired with
TRAIN_FLOPS_CAP=3.0e18 so any implementation at >=20% MFU is FLOPs-bound
(4.97 h at 20%, 3.98 h at 25% on 4x5090 = 838 TFLOPS peak bf16), i.e. the
kernel lottery stops being scored for essentially the whole field.
- POD_LIFETIME_HOURS_CAP: 7.0 and 8.5 were both defensible and mutually
wrong. 7.0 ("6h train + 1h") was already broken by the harness's own
phase ceilings -- the train child alone is 6.78 h at a 6 h cap and the
eval child gets its whole timeout on top, so a full-budget run could be
killed mid-eval and lose the entire miner-funded rental; a prior pass
raised it to 8.5, which is right FOR A 6 H TRAIN CAP. With the train cap
at 5.0 h the arithmetic closes at 7.5 h:
train child = 900 build + 18000 train + 120 grace + 1800 ckpt = 20820 s
eval child = 5400 (battery 3600 + load/rollup/score reserve)
worst case = 26220 s = 7.28 h => 7.5 h leaves 780 s
A new assertion requires the cap to be TIGHT (pod - worst_case < 1 h), so
padding it instead of re-deriving it fails the test.
- prism_lium_payer::sealed hardcoded 6 h train + 2 h eval, which is how the
payer came to model an 8 h pod against a 7 h recipe cap. TRAIN_WALL_SECS
and EVAL_BUDGET_SECS are now DERIVED from the recipe constants and a test
asserts they reconstruct POD_LIFETIME_HOURS_CAP exactly and still cover
HARNESS_EVAL_TIMEOUT_S. The payer cannot silently drift again.
- EVAL_GLOBAL_BUDGET_S mirrors eval.common.BATTERY_BUDGET_S, asserted
against the Python string, so the pod arithmetic is not computed on a
fiction.
MAX_PARAMS stays 1e9. Under iso-FLOPs it is NON-BINDING -- the compute
optimum is around N_body ~ 143M and the 0.02-nat plateau spans ~88-236M
(2.7x) -- so it is a VRAM/checkpoint parameter, not a scientific one.
finalize.rs: reads budget.flops_attested / budget.binding_cap from the
harness `budget` block, gated on the SAME tokens_seen_source ==
"train_stream" discriminator as tokens. flops_attested is
flops_per_token * tokens_seen, so an unauthoritative token count makes the
product unauthoritative; treating a bypassed-stream run's FLOPs as attested
would let a miner with their own dataloader choose their own budget.
Also fixes a latent scoring bypass in the same reader: submission_metrics
used `?` on every org.* entry, so ONE unparseable value made it return
None -- which SKIPS the composite silently. A typo, or a harness emitting a
string diagnostic into the battery, was enough. Now unparseable entries are
skipped: if the key is declared it becomes a hard MissingMetric ->
Ineligible (fail-closed), and if it is not declared it was inert anyway.
api.rs (not owned here, minimal change): the /v1/recipe assertion tracked
a hardcoded 6.0, so it now tracks the constant and additionally asserts the
currency and the underspend floor are advertised -- both are miner-facing.v3 = v2 plus the measurement changes of the dual-cap design. Placeholder only: LATEST_ANCHOR_VERSION -> 3, DEFAULT_ANCHOR_VERSION STAYS 0, and v0/v1/v2 stay byte-frozen with their pre-registration hashes now pinned explicitly so a "harmless" reformat fails a test instead of an audit. (1) REMOVES org.g8.mup_scaling_slope from the scored set. For L = E + A/N^alpha the measured local slope is alpha*(1 - E/L) -- only ~30-56% of alpha over the plausible E/L range -- so a model that is better IN LEVEL can look like it scales WORSE. The natural repair (score the growth of the advantage over a fixed reference, which cancels E exactly) also fails, but on noise rather than bias: OLS over 4 log-spaced rungs with 3 seeds and a reused reference gives an MDD of 0.019-0.028 against a plausible architectural signal of 0.013-0.065, and at sigma_lnL = 0.05 the MDD is 0.055 and the metric is dead. The harness KEEPS EMITTING the key -- a key absent from the anchor set is inert, not missing, which is exactly "telemetry, never scored". A test asserts both halves: gone from v3, still emitted by eval/g8_stability.py. (2) ADDS three G6 keys, re-reading G6 as data AND COMPUTE efficiency: auc_log_bytes, bytes_to_bpb_threshold, bpb_at_half_budget. auc_log_bytes supersedes auc_log_tokens because per-token CE is not tokenizer-neutral -- a tokenizer that compresses harder lowers CE/token without predicting better. That rename is only HONEST because the probe contract now records bytes: a test asserts the harness carries bytes_seen / bytes_per_token / probe_bits_per_byte, so the key cannot name a quantity that does not exist. bpb_at_half_budget reads an ORGANIZER compute milestone (0.5*C_MAX) rather than the existing miner-controlled report counter. Direction is asserted: cap < reference is the ONLY encoding of lower-better, which is the bug the v0 auc key got backwards. (3) ADDS the confirmation tier (org.conf.*) as a 5-rung IsoFLOP mini-profile for finalists, under a group named `conf` at weight 0. That group is STRUCTURALLY inert, not merely down-weighted: composite hardcodes GROUP_KEYS = [g1..g8] with [f64; N_GROUPS] arrays, so a group outside that list is never validated, normalized, or required for completeness -- which is what makes the tier a separate audit record. Deliberately NOT named g9, because a g9-looking group would read as a scored group that is silently ignored. Scored: isoflop_min_bpb (the level at the profile minimum -- E-free, SE ~ 0.01-0.03 nats, and the right level to score, unlike a fixed-D ladder's top rung which is the most DATA-STARVED point) and isoflop_convexity_r2. Observed-only: isoflop_argmin_nbody (the argmin is the LEAST identifiable feature by construction, because the minimum is flat -- +-17% in N at sigma=0.02, +-47% at 0.05) and advantage_growth. Both never-scored keys carry a DEGENERATE normalizer (reference == cap), so promoting them into a scored group collapses the geometric mean to 0 rather than quietly ranking noise -- an unmissable failure instead of a plausible-looking one. (4) ADDS the compute gates max_flops = 3.0e18 and min_spend_fraction = 0.5, with max_wall_s 21600 -> 18000 since wall-clock is now only the anti-DoS bound. Asserted to track the recipe constants rather than repeat them, and asserted to be the ONLY gate differences from v2. max_params unchanged. Compute-optimal-normalized scoring is deliberately NOT the scored surface: it creates a deliberate-underspend attack (frontier slope only ~-0.05..-0.10 nats/e-fold). The frontier is for anchor reference values, a truncation correction bounded at <= 0, and published diagnostics. Every v3 numeric is `placeholder` and every note states the measurement obligation -- asserted by a test, so a value cannot be quietly promoted to measured. v3 must not be SELECTED until the harness emits every key it declares (org.conf.* needs the operator tier; org.g6.* needs the eval/g6_curve.py byte-curve change, which is sibling-owned): a declared but absent key is a hard MissingMetric -> Ineligible -> lattice 0 for every in-flight submission. Emit before declare.
docs/PRISM_RECIPE.md -- both cap tables were stale in two directions: they still advertised a 6.0 h wall, a 7.0 h pod and a 350M param cap, the last of which the prose two sections below already contradicted with 1B. Now: the currency (3.0e18 attested FLOPs), the wall as a safety bound (5.0 h), the underspend floor, the derived 7.5 h pod lifetime with the arithmetic shown, the global eval battery, and 1B params. New section on the currency covering what the attestation catches (probe detection via secret indices and the max-not-median switch, stream bypass via tokens_seen_source, physically-impossible claims) and what it CANNOT: FlopCounterMode sees only what the dispatcher sees, so an opaque fused kernel is invisible, and recipe-v10 makes that reachable. The analytic cross-check and its 25% threshold are documented as review EVIDENCE, not as a fail -- a genuinely novel architecture can widen the gap honestly. Also documents why C = 6ND is unusable here (the lm_head is ~36% of FLOPs/token at d=512, so 6ND overstates the affordable tokens by 1.3-1.8x) and states plainly that the 20% MFU margin is TIGHT: at 15% real MFU a full budget needs 6.63 h and the wall binds again, which is why MFU is measured rather than assumed. deploy/scripts/prism-phase0-seed-variance.sh (new) measures the two quantities the whole design is gated on: - sigma_seed. The existing clustered bootstrap resamples EVAL ITEMS only and has never measured TRAINING variance. A seed change alone re-ranks NAS architectures at Kendall tau = 0.48, so the LCB is overconfident: it charges a submission for the noise it does measure and gives away the noise it does not. If sigma_seed exceeds typical inter-architecture gaps, no emission rule can save the ranking and runs must be seed-averaged -- which changes cost per submission, hence the subnet's economics. - real MFU. Every FLOPs constant scales linearly with it. The seed is a HARNESS env knob (PRISM_SEED_OVERRIDE), never a submission field: the API has no env passthrough and must not grow one, because a miner-chosen seed would make two submissions incomparable. The script therefore emits a per-seed compose override and runs sequentially, and the manifest records the knob so a run trained off the lattice seed is visible in its own attestation rather than inferred. Spending discipline is built in: --dry-run validates the whole wiring (baselines on disk, Lium key, /health, that the DEPLOYED build actually advertises the dual cap, the harness markers, and the reduction math via a static stats self-test) and rents nothing; without --confirm-spend it refuses to launch and prints the estimate. The dry-run report is written from deterministic pseudo-values and stamped SYNTHETIC so it cannot be mistaken for a measurement. Reduction reports mean/sigma/SE/CI per metric, states that the normal approximation UNDERSTATES the interval at n=3 (t_2,.975 = 4.30 vs 1.96) instead of hiding it, and notes the 95% interval on sigma itself spans ~0.52x-3.7x at n=3 -- an order of magnitude, not a calibration constant. It also verdicts FLOPS-BOUND vs WALL-BOUND against the 5.0 h wall, and checks the counter-vs-analytic gap on a KNOWN-HONEST baseline, which is the precondition for treating a wide gap on a real submission as evidence.
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Every real pod run on this branch died at import: File "/tmp/prism_eval/main.py", line 74, in <module> from prismlib import deps as deps_mod ImportError: cannot import name 'deps' from 'prismlib' `prismlib/deps.py` was added in faac738 (recipe-v10 miner dep install) and imported at MODULE SCOPE in main.py, but never added to `HARNESS_FILES` -- which is the upload manifest. A module absent from that list does not exist on the pod, however normal it looks on disk. Found by an actual Lium run, not by review. Why no test caught it: local tests import prismlib from the SOURCE TREE, where deps.py is present, so every harness test passed. The existing coverage is a hardcoded list of expected paths, and a new import with no matching entry is exactly the thing nobody remembers to add to it. So the fix is two parts: - add the missing include_str! entry (and to the expected-path list), and - add `every_imported_prismlib_module_is_uploaded`, which parses every `from prismlib import x` / `from prismlib.x import y` in every uploaded .py and asserts a matching `prismlib/x.py` is in HARNESS_FILES. Verified to FAIL with the entry removed and pass with it. It also asserts the scan found >= 10 imports, so if the parser stops matching the test reports that instead of silently guarding nothing. Scope: prism-v2.1-scoring and everything stacked on it. `main` does NOT import deps, so live is unaffected. The sibling branch working on crates/prism-recipe/harness/eval/** is affected and should take this fix. This moves `recipe_pin_hex()` (harness bytes are hashed) -- unavoidable, since the alternative is a harness that cannot start.
`harness_env_pairs` is an ALLOWLIST, and the dual-cap knobs were not on it, so none of them reached the pod over SSH. Two consequences, one of which would have produced a confidently wrong measurement rather than a visible failure: 1. PRISM_SEED_OVERRIDE was dropped. The Phase-0 seed-variance sweep sets the seed on the control plane, the pod never saw it, and every "different seed" run would have trained on the SAME lattice seed -- reporting sigma_seed ~ 0 and "clearly safe to enable significance gating", which is the exact opposite of what the measurement exists to establish. A silently-ignored knob is worse than a missing feature here. 2. PRISM_TRAIN_FLOPS_CAP / PRISM_MIN_SPEND_FRACTION were left to the pod's compiled defaults. They are now sent EXPLICITLY from the recipe constants, the same way the wall cap already was, so the pod enforces the master's budget rather than whatever default its harness copy happens to carry. Also forwards PRISM_TEST_TRAIN_FLOPS (reduced-budget measurement waves) and the probe knobs PRISM_FLOPS_PROBE_SAMPLES / _CV_MAX / _ANALYTIC_GAP_MAX. All of them pass the existing numeric guard, so a shell payload in a forwarded value is DROPPED rather than quoted and hoped for -- asserted with the same `rm -rf` probe the other knobs use. The new test also asserts the measurement-only knobs are ABSENT by default, so knowing about them cannot leak them into a scored round. phase-0 script: applying the seed depends on how the control plane is deployed, since the knob is read from the CONTROL PLANE's process environment. Added PRISM_PHASE0_SEED_HOOK for operators not running compose (a bare process, or another supervisor); without a hook the previous compose-override path is unchanged. The hook must fail loudly, for the same reason as (1): a hook that silently no-ops reproduces the sigma_seed ~ 0 failure this commit exists to prevent.
Four of G2's eight sub-metrics normalize to a constant 0 for the entire field at this operating point. Because G2's sub-metrics are equal-weighted they carried half of G2's weight -- 7.5% of the whole composite -- while measuring nothing, and because the composite is a weighted GEOMETRIC mean an axis pinned at 0 is actively harmful rather than merely inert. Drop arc_challenge, winogrande, boolq and obqa; keep LAMBADA-strict, HellaSwag, PIQA and ARC-easy at equal weight. The G2 group weight stays 0.15: this changes what G2 measures, not how much G2 counts. Each survivor's share of G2 doubles to 0.25 (3.75% of the composite). Not a cap problem: separating two submissions on Winogrande needs ~76824 items and the set has 1267, so the resolution does not exist at any cap. Retire rather than demote to weight 0.0, because a declared metric is still measured and measurement nobody is paid for is pod time. Runs scored against v3 must set PRISM_EVAL_G2_TASKS; that harness support lands on the stacked prism-v3-incentives branch, so both land together.
The probe adds a fwd+bwd of its own on top of a model already resident on
the GPU, so it can OOM where training itself would not -- observed on the
hybrid delta-net baseline at 341M params on a 32 GB card.
A probe that merely fails leaves the FLOPs cap DISARMED, which is worse
than it looks: it loses attestation on exactly the memory-heavy
architectures where the budget matters most, and it is an escape a miner
could induce deliberately, since OOM-ing the probe converts the budget
back to wall-clock.
So an OOM now halves the probe's row count and retries, down to a single
sequence, freeing the CUDA cache between attempts. That stays a valid
measurement because the quantity is FLOPs PER TOKEN at a fixed sequence
length -- trimming rows changes efficiency, not cost per token -- and the
test asserts the reduced-row result agrees with the full-batch one.
probe_rows / probe_rows_reduced are reported and emitted as
org.diag.flops_probe_rows{,_reduced} so a reduced-batch attestation is
never silently equated with a full-batch one. Only OOM degrades: any
other exception still propagates rather than being retried into a wrong
number.Measured on 1xRTX 5090, reference Transformer++, three seeds, seed varied alone, at a REDUCED 3.0e17 train budget (~6% of TRAIN_FLOPS_CAP) to stay inside the spend ceiling. sigma_seed is therefore a LOWER BOUND and is labelled as one throughout. Real MFU = 40.2% (sd 0.0016). The design needed >=20% to stay FLOPs-bound in the 5.0h wall and feared 15%, so the "design reverts to the status quo" risk is retired: a full budget needs ~2.5h on 4 GPUs. The analytic cross-check agrees with the dispatcher counter to 1.1% (ratio 1.0113, threshold 25%), probe CV is exactly 0, and bytes_seen x F_tok reproduces flops_attested to 0.00%. sigma_seed splits sharply by item count, not by architecture: G1 bits/byte is stable (prose 0.0123 bpb, 0.61% CV) while 1-2 cluster accuracy metrics swing 43-173% CV. The decisive finding is a gate, not a variance: the hard G3 floor FLIPPED on seed alone (seed1001 g3=0.198 < 0.25 -> composite forced to 0, seeds 1002/1003 passed), driven by passkey_acc 0.0->0.5 on one cluster. A binary gate does not degrade with variance, it flips -- so the significance-gated emission rule cannot be armed yet even though continuous G1 ranking is sound. All three runs were ineligible for an unrelated reason (four keys the battery does not emit), so the composite spread is a gate artifact and composite sigma_seed remains unmeasured. Also assert in prism-recipe that the step cap and the FLOPs cap are only mutually reachable at a large batch: at the reference batch 8 x seq 512 the 20000-step cap buys 8.2e7 tokens = 1.8e17 FLOPs, only ~6% of the cap, so MIN_SPEND_FRACTION would mark the reference baseline itself Ineligible. Reaching the cap needs batch ~132 at seq 512.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
prism-v2.1-scoring. Do not merge tomain.Replaces the fixed wall-clock training budget with a defensible compute
currency, and replaces the confounded scaling-slope metric with a
measurement programme.
DEFAULT_ANCHOR_VERSIONstays0, so live scoringis bit-identical — this is pre-registration, not a governance flip.
The currency: attested FLOPs, dual-capped
TRAIN_FLOPS_CAP = 3.0e18becomes the budget;TRAIN_HOURS_CAPdrops6.0 → 5.0 h and is demoted to an anti-DoS bound. Whichever cap binds first
stops the run, and
org.diag.binding_caprecords which one did. The minerpicks
NandDfreely underneath.Wall-clock as a currency makes MFU a scored quantity: two identical
architectures differ in score by kernel maturity, and a looped model at
r=4pays ~3.3× FLOPs/token so it sees ~3.3× fewer tokens — charged to thearchitecture as if it were a defect. FLOPs measured from the realized
dispatch graph price looping, MoE sparsity and vocabulary automatically, so
the budget is class-adaptive with no tier to declare and none to shop
for. Size tiers were considered and rejected: if
C_kgrows with thetier then loss is monotone in tier, the interior optimum disappears, the
tier boundary becomes the new cap, and tier-shopping becomes rational
rather than an exploit — and tiers are incoherent with winner-take-all
emission.
Attestation is harness-owned end to end; the miner reports no number:
Enforcement lives in
SeededTrainStream.next_batch, which refuses toyield past a cap — replacing a cooperative
ctx["guard"]closure whose_CapExceededwas caught by nothing and failed the whole run. Reaching yourbudget is now the expected outcome and routes to the graceful
checkpoint-then-eval path.
The hole this cannot close by counting
FlopCounterModeonly sees what the PyTorch dispatcher sees, and recipe-v10lets miners install their own dependencies — so a fused Triton/CUDA kernel
registered as one opaque dispatch is invisible. Mitigated by an analytic
cross-check from the module graph:
C = 6NDis wrong at this scale: atd=512, V=32768thelm_headalone is~36% of FLOPs/token, so
6·N_bodycaptures only ~55% of the cost and6NDoverstates affordable tokens by 1.3–1.8×. MoE counts active experts;
only the body loops (hence 3.3×, not 4×); the quadratic attention term is
charged only when attention is detected, so a delta-net/SSM is not
billed for a phantom cost. A gap > 25% is published as
org.diag.flops_analytic_ratio/_gapand flagged — review evidence,never a silent pass, because a genuinely novel architecture can widen the
gap honestly.
Budget reconciliation (7.0 vs 8.5 h)
Both prior values were defensible and mutually wrong. 7.0 h ("6h + 1h") was
already broken by the harness's own phase ceilings; 8.5 h is correct for a
6 h train cap; the payer separately hardcoded 6h+2h = 8h. With a 5.0 h
train cap the arithmetic closes at 7.5 h:
prism_lium_payer::sealednow derives its TTL from these constants and atest asserts it reconstructs the pod cap exactly, so the payer cannot drift
again. A further assertion requires the cap to be tight (
pod − worst < 1h),so padding it fails instead of hiding a future over-subscription.
Anchors v3 (placeholder, unselected)
org.g8.mup_scaling_slopefrom the scored set. The measuredlocal slope is
α·(1 − E/L)≈ 30–56% ofα, so a model better in levellooks like it scales worse. The E-cancelling repair also fails, on noise:
MDD 0.019–0.028 vs a signal of 0.013–0.065. Still emitted — inert
telemetry, exactly "never scored".
org.g6.auc_log_bytes,bytes_to_bpb_threshold,bpb_at_half_budget. The byte rename is honest only because the probecontract now records bytes — asserted by a test rather than assumed.
org.conf.*confirmation tier (5-rung IsoFLOP profile,finalists only) under a
confgroup at weight 0 — structurally inert,since
GROUP_KEYSis[g1..g8]. Deliberately not namedg9. The twonever-scored keys carry a degenerate normalizer so accidental promotion
collapses the composite to 0 rather than ranking noise.
max_flops+min_spend_fractiongates;max_wall_s21600 → 18000.
v0/v1/v2stay byte-frozen, with their prereg hashes now pinnedexplicitly.
DEFAULT_ANCHOR_VERSIONstays0.Compute-optimal-normalized scoring is deliberately not the scored
surface — it invites deliberate underspend (frontier slope only
−0.05..−0.10 nats/e-fold). Hence
MIN_SPEND_FRACTION = 0.5as aneligibility floor, and any truncation correction bounded at ≤ 0.
Two latent bugs fixed in passing
submission_metricsused?on everyorg.*entry,so one unparseable value made it return
None— which skips thecomposite silently. A typo was enough. Now skipped per-entry: declared
keys fail closed as
MissingMetric→Ineligible.the composite reads
org.*from the battery the eval child writes, so thewhole diagnostic set was emitted and discarded. Merged now — numeric keys
only (a string would break the reader), and never creating a battery (a
diag-only battery would turn "composite skipped" into
Ineligible).G2: four pinned sub-metrics retired from anchors v3
Four of G2's eight sub-metrics normalize to a constant 0 for the entire
field at this operating point. G2's sub-metrics are equal-weighted, so
arc_challenge,winogrande,boolqandobqacarried half of G2's0.15 weight — 7.5 % of the whole composite — while measuring nothing. And
because the composite is a weighted geometric mean, an axis pinned at 0
is actively harmful rather than merely inert.
Dropped; LAMBADA-strict / HellaSwag / PIQA / ARC-easy survive at equal
weight, each doubling to 0.25 of G2 (3.75 % of the composite). G2 group
weight is unchanged at 0.15 — this changes what G2 measures, not how
much G2 counts.
Not a cap problem: separating two submissions on Winogrande needs ~76 824
items and the set has 1 267, so the resolution does not exist at any cap.
Retired rather than demoted to
weight: 0.0, because a declared metric isstill measured, and measurement nobody is paid for is pod time.
Dependency: runs scored against v3 must set
PRISM_EVAL_G2_TASKS=lambada,hellaswag,piqa,arc_easy. That harness support(
eval_g2_tasks()+ per-taskmean_gold_nll) lands on the stackedprism-v3-incentivesbranch (#168), so both branches must landtogether. Do not set it for v0/v1/v2, which declare all eight keys and
would fail their own completeness gate.
Phase 0: σ_seed and real MFU measured
Evidence under
docs/evidence/prism-v3-phase0/(evidence, never normative).Reference Transformer++ (
d=1024, L=24), three seeds, seed varied alone,1×RTX 5090, at a reduced
3.0e17train budget (~6 % of the cap) to stayinside the spend ceiling — so σ_seed here is an explicit lower bound.
Real MFU = 40.2 % (sd 0.0016). The design needed ≥ 20 % to stay
FLOPs-bound in the 5.0 h wall and feared 15 %; a full budget needs ≈ 2.5 h on
4 GPUs. The "design reverts to the status quo" risk is retired.
Attestation validated against hardware: the analytic cross-check agrees
with the dispatcher counter to 1.1 % (ratio 1.0113, flag threshold 25 %),
probe CV is exactly 0, and
bytes_seen × F_tokreproducesflops_attestedto 0.00 %.σ_seed splits by item count, not architecture: G1 bits/byte is stable
(prose 0.0123 bpb, 0.61 % CV) while 1–2-cluster accuracy metrics swing
43–173 % CV.
The decisive finding is a gate, not a variance. The hard G3 floor
flipped on seed alone — seed 1001 scored
g3 = 0.198 < 0.25and was forcedto composite 0 while seeds 1002/1003 passed, driven by
passkey_accswinging 0.0 → 0.5 on a single cluster. A binary gate does not degrade
with variance, it flips. So continuous G1 ranking is sound at this budget
(0.0123 bpb sits below the 0.02-nat plateau the competition must resolve),
but the significance-gated emission rule on #168 cannot be armed yet.
All three runs were
ineligiblefor an unrelated reason — four keys thebattery does not emit at this recipe version — so the 0.0-vs-0.330 composite
spread is a gate artifact, not composite σ_seed, which remains
unmeasured.
Also asserted: the step cap and the FLOPs cap are only mutually reachable at
a large batch. At the reference
batch 8 × seq 512,MAX_TRAIN_STEPSbuys
8.2e7tokens =1.8e17FLOPs, only ~6 % of the cap — soMIN_SPEND_FRACTIONwould mark the reference baseline itselfIneligible.Reaching the cap needs batch ≈ 132.
Gates
cargo fmt·clippy -D warnings·cargo test --workspace(allgreen, 0 failed) ·
cargo deny·loc-cap·consensus-lint·spec-check·design-check·external-docs-check— all green. Every harness Pythontest passes, including
smoke_battery.py(BATTERY SMOKE OK) and 23 newattestation cases.
prism-budgetwas split out ofprism-pipeline::compositefor the 1500non-test LOC cap (precedent:
prism-competitionout ofprism-registry);every type is re-exported, so the public API and JSON shape are unchanged.
Not in this PR
RECIPE_VERSIONstays2.0.0. The pin (recipe_pin_hex()) moves becauseharness bytes are hashed, but the semver bump to 2.1.0 is a release action
with miner-facing docs, the public
BaseIntelligence/prismrepo and theexternal-docs-checkpin attached — same precedent as theMAX_PARAMSraise. Sibling-owned changes needed before v3 can be selected are listed
in the handover report (
eval/g6_curve.pybyte curve,eval/g9_isoflop.py,the ≥300-step G8 sweep,
docs/PRISM.md+docs/external-miner/).