M19.3/M19.5/M19.7: the three training gates become functions by codeitlikemiley · Pull Request #40 · codeitlikemiley/panday

codeitlikemiley · 2026-08-24T00:09:35Z

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is exactly what M19.1 was written against. The models are still absent and that does not change; what changes is that when one arrives, there is something to judge it with.

No fabricated scores. The gates ship, not results.

M19.3 — `Score::beats(&incumbent, 10.0)`

The milestone's bar is relative ("beats heuristic by ≥10pt"); the existing meets_the_gate is an absolute floor answering a different question.

The margin is percentage points, not a fraction. Passing 0.10 and meaning ten points is the obvious way to get this wrong, both readings compile and both look right, so a test pins the unit.
A confidently-wrong challenger is refused regardless of margin. The dangerous quadrant is what route-bench exists to measure. A caller checking only the margin would ship a model that is better on average and catastrophic on the cases where the router acts on the answer.

M19.5 — `Report::cost_saving` / `cheaper_than(0.25, …)`

reduce-bench had no cost dimension at all. It scored retention and token ratio, which are not money — so "≥25% cheaper" was unmeasurable even with a model in hand.

Cost rather than tokens, because the two move independently: a reducer that keeps 40% of the tokens and hands them to a model at 3× the price is more expensive, and the old ratio-only scorecard would have called that a 60% win.

Prices are per token and supplied by the caller — panday-harness has no price table and must not grow one, or the gate starts drifting with a catalog. A saving against an incumbent that spends nothing returns None rather than a number. Both halves required: cheap-with-regressions is a reducer that saves money by losing the answer.

M19.7 — `Scorecard::beats(&base)`

On the artifact type every suite already emits, so it isn't agent-bench's alone.

It returns Option<bool>, and the None is the substance. Comparing a tuned run over 10 tasks against a base run over 41 is how this measurement gets faked without anyone lying: both numbers are real, the ratio is meaningless, and a plain bool would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. .unwrap_or(false) — what a caller will actually write — fails closed.

Falsified

Injected defect	Result
M19.3 margin treated as a fraction	red — unit test
M19.5 prices both sides identically	red — cost test
M19.7 case-count guard removed	red — two tests, incl. fail-closed

Green after restoring: 34/34 router, 42/42 harness, 13/13 types.

Numbering

No new milestone numbers — these are the gates of M19.3/5/7. Each bullet in docs/19 now records that its gate is executable while its model is not.

Verification

fmt · clippy --workspace --all-targets -D warnings · 1114 workspace tests, 0 failed · schemas · ts-sdk · sbom (609 components) · deny.

https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is precisely what M19.1 was written against. The models are still absent and that is unchanged; what changes is that when one arrives there is something to judge it with. **M19.3 — `Score::beats(&incumbent, 10.0)`.** The milestone's bar is relative ("beats heuristic by ≥10pt"); `meets_the_gate` is an absolute floor and answers a different question. The margin is percentage points, not a fraction: passing `0.10` and meaning ten points is the obvious way to get this wrong, both readings compile, so a test pins the unit. A confidently-wrong challenger is refused regardless of margin — the dangerous quadrant is what route-bench exists to measure, and a caller checking only the margin would ship a model that is better on average and catastrophic where the router acts on it. **M19.5 — `Report::cost_saving` and `cheaper_than(0.25, ..)`.** reduce-bench had **no cost dimension at all**: it scored retention and token ratio, which are not money, so "≥25% cheaper" was unmeasurable with a model in hand. Cost rather than tokens because the two move independently — a reducer keeping 40% of the tokens and handing them to a model at 3x the price is *more* expensive, and the old scorecard would have called that a 60% win. Prices are per token and supplied by the caller; this crate has no price table and must not grow one, or the gate drifts with a catalog. A saving against an incumbent that spends nothing is `None`, not a number. Both halves are required: cheap-with-regressions is a reducer that saves money by losing the answer. **M19.7 — `Scorecard::beats(&base)`**, on the artifact type every suite emits. It returns `Option<bool>` and the `None` is the substance: comparing a tuned run over ten tasks against a base over forty-one is how this gets faked without anyone lying — both numbers real, ratio meaningless — and a plain `bool` would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. `.unwrap_or(false)` fails closed. No new milestone numbers: these are the gates *of* M19.3/5/7, so each bullet records that its gate is now executable while its model is not. All three falsified. Making the margin a fraction reddens the unit test; pricing both sides of reduce-bench identically reddens the cost test; removing the case-count guard reddens two, including the fail-closed one. Green after restoring. Verified: fmt, clippy -D warnings, 1114 workspace tests, schemas, ts-sdk, sbom, deny. Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

codeitlikemiley · 2026-08-24T00:09:35Z

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is exactly what M19.1 was written against. The models are still absent and that does not change; what changes is that when one arrives, there is something to judge it with.

No fabricated scores. The gates ship, not results.

M19.3 — `Score::beats(&incumbent, 10.0)`

The milestone's bar is relative ("beats heuristic by ≥10pt"); the existing meets_the_gate is an absolute floor answering a different question.

The margin is percentage points, not a fraction. Passing 0.10 and meaning ten points is the obvious way to get this wrong, both readings compile and both look right, so a test pins the unit.
A confidently-wrong challenger is refused regardless of margin. The dangerous quadrant is what route-bench exists to measure. A caller checking only the margin would ship a model that is better on average and catastrophic on the cases where the router acts on the answer.

M19.5 — `Report::cost_saving` / `cheaper_than(0.25, …)`

reduce-bench had no cost dimension at all. It scored retention and token ratio, which are not money — so "≥25% cheaper" was unmeasurable even with a model in hand.

Cost rather than tokens, because the two move independently: a reducer that keeps 40% of the tokens and hands them to a model at 3× the price is more expensive, and the old ratio-only scorecard would have called that a 60% win.

Prices are per token and supplied by the caller — panday-harness has no price table and must not grow one, or the gate starts drifting with a catalog. A saving against an incumbent that spends nothing returns None rather than a number. Both halves required: cheap-with-regressions is a reducer that saves money by losing the answer.

M19.7 — `Scorecard::beats(&base)`

On the artifact type every suite already emits, so it isn't agent-bench's alone.

It returns Option<bool>, and the None is the substance. Comparing a tuned run over 10 tasks against a base run over 41 is how this measurement gets faked without anyone lying: both numbers are real, the ratio is meaningless, and a plain bool would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. .unwrap_or(false) — what a caller will actually write — fails closed.

Falsified

Injected defect	Result
M19.3 margin treated as a fraction	red — unit test
M19.5 prices both sides identically	red — cost test
M19.7 case-count guard removed	red — two tests, incl. fail-closed

Green after restoring: 34/34 router, 42/42 harness, 13/13 types.

Numbering

No new milestone numbers — these are the gates of M19.3/5/7. Each bullet in docs/19 now records that its gate is executable while its model is not.

Verification

fmt · clippy --workspace --all-targets -D warnings · 1114 workspace tests, 0 failed · schemas · ts-sdk · sbom (609 components) · deny.

https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is precisely what M19.1 was written against. The models are still absent and that is unchanged; what changes is that when one arrives there is something to judge it with. **M19.3 — `Score::beats(&incumbent, 10.0)`.** The milestone's bar is relative ("beats heuristic by ≥10pt"); `meets_the_gate` is an absolute floor and answers a different question. The margin is percentage points, not a fraction: passing `0.10` and meaning ten points is the obvious way to get this wrong, both readings compile, so a test pins the unit. A confidently-wrong challenger is refused regardless of margin — the dangerous quadrant is what route-bench exists to measure, and a caller checking only the margin would ship a model that is better on average and catastrophic where the router acts on it. **M19.5 — `Report::cost_saving` and `cheaper_than(0.25, ..)`.** reduce-bench had **no cost dimension at all**: it scored retention and token ratio, which are not money, so "≥25% cheaper" was unmeasurable with a model in hand. Cost rather than tokens because the two move independently — a reducer keeping 40% of the tokens and handing them to a model at 3x the price is *more* expensive, and the old scorecard would have called that a 60% win. Prices are per token and supplied by the caller; this crate has no price table and must not grow one, or the gate drifts with a catalog. A saving against an incumbent that spends nothing is `None`, not a number. Both halves are required: cheap-with-regressions is a reducer that saves money by losing the answer. **M19.7 — `Scorecard::beats(&base)`**, on the artifact type every suite emits. It returns `Option<bool>` and the `None` is the substance: comparing a tuned run over ten tasks against a base over forty-one is how this gets faked without anyone lying — both numbers real, ratio meaningless — and a plain `bool` would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. `.unwrap_or(false)` fails closed. No new milestone numbers: these are the gates *of* M19.3/5/7, so each bullet records that its gate is now executable while its model is not. All three falsified. Making the margin a fraction reddens the unit test; pricing both sides of reduce-bench identically reddens the cost test; removing the case-count guard reddens two, including the fail-closed one. Green after restoring. Verified: fmt, clippy -D warnings, 1114 workspace tests, schemas, ts-sdk, sbom, deny. Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

codeitlikemiley · 2026-08-24T00:09:35Z

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is exactly what M19.1 was written against. The models are still absent and that does not change; what changes is that when one arrives, there is something to judge it with.

No fabricated scores. The gates ship, not results.

M19.3 — `Score::beats(&incumbent, 10.0)`

The milestone's bar is relative ("beats heuristic by ≥10pt"); the existing meets_the_gate is an absolute floor answering a different question.

The margin is percentage points, not a fraction. Passing 0.10 and meaning ten points is the obvious way to get this wrong, both readings compile and both look right, so a test pins the unit.
A confidently-wrong challenger is refused regardless of margin. The dangerous quadrant is what route-bench exists to measure. A caller checking only the margin would ship a model that is better on average and catastrophic on the cases where the router acts on the answer.

M19.5 — `Report::cost_saving` / `cheaper_than(0.25, …)`

reduce-bench had no cost dimension at all. It scored retention and token ratio, which are not money — so "≥25% cheaper" was unmeasurable even with a model in hand.

Cost rather than tokens, because the two move independently: a reducer that keeps 40% of the tokens and hands them to a model at 3× the price is more expensive, and the old ratio-only scorecard would have called that a 60% win.

Prices are per token and supplied by the caller — panday-harness has no price table and must not grow one, or the gate starts drifting with a catalog. A saving against an incumbent that spends nothing returns None rather than a number. Both halves required: cheap-with-regressions is a reducer that saves money by losing the answer.

M19.7 — `Scorecard::beats(&base)`

On the artifact type every suite already emits, so it isn't agent-bench's alone.

It returns Option<bool>, and the None is the substance. Comparing a tuned run over 10 tasks against a base run over 41 is how this measurement gets faked without anyone lying: both numbers are real, the ratio is meaningless, and a plain bool would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. .unwrap_or(false) — what a caller will actually write — fails closed.

Falsified

Injected defect	Result
M19.3 margin treated as a fraction	red — unit test
M19.5 prices both sides identically	red — cost test
M19.7 case-count guard removed	red — two tests, incl. fail-closed

Green after restoring: 34/34 router, 42/42 harness, 13/13 types.

Numbering

No new milestone numbers — these are the gates of M19.3/5/7. Each bullet in docs/19 now records that its gate is executable while its model is not.

Verification

fmt · clippy --workspace --all-targets -D warnings · 1114 workspace tests, 0 failed · schemas · ts-sdk · sbom (609 components) · deny.

https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is precisely what M19.1 was written against. The models are still absent and that is unchanged; what changes is that when one arrives there is something to judge it with. **M19.3 — `Score::beats(&incumbent, 10.0)`.** The milestone's bar is relative ("beats heuristic by ≥10pt"); `meets_the_gate` is an absolute floor and answers a different question. The margin is percentage points, not a fraction: passing `0.10` and meaning ten points is the obvious way to get this wrong, both readings compile, so a test pins the unit. A confidently-wrong challenger is refused regardless of margin — the dangerous quadrant is what route-bench exists to measure, and a caller checking only the margin would ship a model that is better on average and catastrophic where the router acts on it. **M19.5 — `Report::cost_saving` and `cheaper_than(0.25, ..)`.** reduce-bench had **no cost dimension at all**: it scored retention and token ratio, which are not money, so "≥25% cheaper" was unmeasurable with a model in hand. Cost rather than tokens because the two move independently — a reducer keeping 40% of the tokens and handing them to a model at 3x the price is *more* expensive, and the old scorecard would have called that a 60% win. Prices are per token and supplied by the caller; this crate has no price table and must not grow one, or the gate drifts with a catalog. A saving against an incumbent that spends nothing is `None`, not a number. Both halves are required: cheap-with-regressions is a reducer that saves money by losing the answer. **M19.7 — `Scorecard::beats(&base)`**, on the artifact type every suite emits. It returns `Option<bool>` and the `None` is the substance: comparing a tuned run over ten tasks against a base over forty-one is how this gets faked without anyone lying — both numbers real, ratio meaningless — and a plain `bool` would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. `.unwrap_or(false)` fails closed. No new milestone numbers: these are the gates *of* M19.3/5/7, so each bullet records that its gate is now executable while its model is not. All three falsified. Making the margin a fraction reddens the unit test; pricing both sides of reduce-bench identically reddens the cost test; removing the case-count guard reddens two, including the fail-closed one. Green after restoring. Verified: fmt, clippy -D warnings, 1114 workspace tests, schemas, ts-sdk, sbom, deny. Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

codeitlikemiley · 2026-08-24T00:09:35Z

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is exactly what M19.1 was written against. The models are still absent and that does not change; what changes is that when one arrives, there is something to judge it with.

No fabricated scores. The gates ship, not results.

M19.3 — `Score::beats(&incumbent, 10.0)`

The milestone's bar is relative ("beats heuristic by ≥10pt"); the existing meets_the_gate is an absolute floor answering a different question.

The margin is percentage points, not a fraction. Passing 0.10 and meaning ten points is the obvious way to get this wrong, both readings compile and both look right, so a test pins the unit.
A confidently-wrong challenger is refused regardless of margin. The dangerous quadrant is what route-bench exists to measure. A caller checking only the margin would ship a model that is better on average and catastrophic on the cases where the router acts on the answer.

M19.5 — `Report::cost_saving` / `cheaper_than(0.25, …)`

reduce-bench had no cost dimension at all. It scored retention and token ratio, which are not money — so "≥25% cheaper" was unmeasurable even with a model in hand.

Cost rather than tokens, because the two move independently: a reducer that keeps 40% of the tokens and hands them to a model at 3× the price is more expensive, and the old ratio-only scorecard would have called that a 60% win.

Prices are per token and supplied by the caller — panday-harness has no price table and must not grow one, or the gate starts drifting with a catalog. A saving against an incumbent that spends nothing returns None rather than a number. Both halves required: cheap-with-regressions is a reducer that saves money by losing the answer.

M19.7 — `Scorecard::beats(&base)`

On the artifact type every suite already emits, so it isn't agent-bench's alone.

It returns Option<bool>, and the None is the substance. Comparing a tuned run over 10 tasks against a base run over 41 is how this measurement gets faked without anyone lying: both numbers are real, the ratio is meaningless, and a plain bool would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. .unwrap_or(false) — what a caller will actually write — fails closed.

Falsified

Injected defect	Result
M19.3 margin treated as a fraction	red — unit test
M19.5 prices both sides identically	red — cost test
M19.7 case-count guard removed	red — two tests, incl. fail-closed

Green after restoring: 34/34 router, 42/42 harness, 13/13 types.

Numbering

No new milestone numbers — these are the gates of M19.3/5/7. Each bullet in docs/19 now records that its gate is executable while its model is not.

Verification

fmt · clippy --workspace --all-targets -D warnings · 1114 workspace tests, 0 failed · schemas · ts-sdk · sbom (609 components) · deny.

https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is precisely what M19.1 was written against. The models are still absent and that is unchanged; what changes is that when one arrives there is something to judge it with. **M19.3 — `Score::beats(&incumbent, 10.0)`.** The milestone's bar is relative ("beats heuristic by ≥10pt"); `meets_the_gate` is an absolute floor and answers a different question. The margin is percentage points, not a fraction: passing `0.10` and meaning ten points is the obvious way to get this wrong, both readings compile, so a test pins the unit. A confidently-wrong challenger is refused regardless of margin — the dangerous quadrant is what route-bench exists to measure, and a caller checking only the margin would ship a model that is better on average and catastrophic where the router acts on it. **M19.5 — `Report::cost_saving` and `cheaper_than(0.25, ..)`.** reduce-bench had **no cost dimension at all**: it scored retention and token ratio, which are not money, so "≥25% cheaper" was unmeasurable with a model in hand. Cost rather than tokens because the two move independently — a reducer keeping 40% of the tokens and handing them to a model at 3x the price is *more* expensive, and the old scorecard would have called that a 60% win. Prices are per token and supplied by the caller; this crate has no price table and must not grow one, or the gate drifts with a catalog. A saving against an incumbent that spends nothing is `None`, not a number. Both halves are required: cheap-with-regressions is a reducer that saves money by losing the answer. **M19.7 — `Scorecard::beats(&base)`**, on the artifact type every suite emits. It returns `Option<bool>` and the `None` is the substance: comparing a tuned run over ten tasks against a base over forty-one is how this gets faked without anyone lying — both numbers real, ratio meaningless — and a plain `bool` would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. `.unwrap_or(false)` fails closed. No new milestone numbers: these are the gates *of* M19.3/5/7, so each bullet records that its gate is now executable while its model is not. All three falsified. Making the margin a fraction reddens the unit test; pricing both sides of reduce-bench identically reddens the cost test; removing the case-count guard reddens two, including the fail-closed one. Green after restoring. Verified: fmt, clippy -D warnings, 1114 workspace tests, schemas, ts-sdk, sbom, deny. Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

codeitlikemiley · 2026-08-24T00:09:35Z

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is exactly what M19.1 was written against. The models are still absent and that does not change; what changes is that when one arrives, there is something to judge it with.

No fabricated scores. The gates ship, not results.

M19.3 — `Score::beats(&incumbent, 10.0)`

The milestone's bar is relative ("beats heuristic by ≥10pt"); the existing meets_the_gate is an absolute floor answering a different question.

The margin is percentage points, not a fraction. Passing 0.10 and meaning ten points is the obvious way to get this wrong, both readings compile and both look right, so a test pins the unit.
A confidently-wrong challenger is refused regardless of margin. The dangerous quadrant is what route-bench exists to measure. A caller checking only the margin would ship a model that is better on average and catastrophic on the cases where the router acts on the answer.

M19.5 — `Report::cost_saving` / `cheaper_than(0.25, …)`

reduce-bench had no cost dimension at all. It scored retention and token ratio, which are not money — so "≥25% cheaper" was unmeasurable even with a model in hand.

Cost rather than tokens, because the two move independently: a reducer that keeps 40% of the tokens and hands them to a model at 3× the price is more expensive, and the old ratio-only scorecard would have called that a 60% win.

Prices are per token and supplied by the caller — panday-harness has no price table and must not grow one, or the gate starts drifting with a catalog. A saving against an incumbent that spends nothing returns None rather than a number. Both halves required: cheap-with-regressions is a reducer that saves money by losing the answer.

M19.7 — `Scorecard::beats(&base)`

On the artifact type every suite already emits, so it isn't agent-bench's alone.

It returns Option<bool>, and the None is the substance. Comparing a tuned run over 10 tasks against a base run over 41 is how this measurement gets faked without anyone lying: both numbers are real, the ratio is meaningless, and a plain bool would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. .unwrap_or(false) — what a caller will actually write — fails closed.

Falsified

Injected defect	Result
M19.3 margin treated as a fraction	red — unit test
M19.5 prices both sides identically	red — cost test
M19.7 case-count guard removed	red — two tests, incl. fail-closed

Green after restoring: 34/34 router, 42/42 harness, 13/13 types.

Numbering

No new milestone numbers — these are the gates of M19.3/5/7. Each bullet in docs/19 now records that its gate is executable while its model is not.

Verification

fmt · clippy --workspace --all-targets -D warnings · 1114 workspace tests, 0 failed · schemas · ts-sdk · sbom (609 components) · deny.

https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is precisely what M19.1 was written against. The models are still absent and that is unchanged; what changes is that when one arrives there is something to judge it with. **M19.3 — `Score::beats(&incumbent, 10.0)`.** The milestone's bar is relative ("beats heuristic by ≥10pt"); `meets_the_gate` is an absolute floor and answers a different question. The margin is percentage points, not a fraction: passing `0.10` and meaning ten points is the obvious way to get this wrong, both readings compile, so a test pins the unit. A confidently-wrong challenger is refused regardless of margin — the dangerous quadrant is what route-bench exists to measure, and a caller checking only the margin would ship a model that is better on average and catastrophic where the router acts on it. **M19.5 — `Report::cost_saving` and `cheaper_than(0.25, ..)`.** reduce-bench had **no cost dimension at all**: it scored retention and token ratio, which are not money, so "≥25% cheaper" was unmeasurable with a model in hand. Cost rather than tokens because the two move independently — a reducer keeping 40% of the tokens and handing them to a model at 3x the price is *more* expensive, and the old scorecard would have called that a 60% win. Prices are per token and supplied by the caller; this crate has no price table and must not grow one, or the gate drifts with a catalog. A saving against an incumbent that spends nothing is `None`, not a number. Both halves are required: cheap-with-regressions is a reducer that saves money by losing the answer. **M19.7 — `Scorecard::beats(&base)`**, on the artifact type every suite emits. It returns `Option<bool>` and the `None` is the substance: comparing a tuned run over ten tasks against a base over forty-one is how this gets faked without anyone lying — both numbers real, ratio meaningless — and a plain `bool` would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. `.unwrap_or(false)` fails closed. No new milestone numbers: these are the gates *of* M19.3/5/7, so each bullet records that its gate is now executable while its model is not. All three falsified. Making the margin a fraction reddens the unit test; pricing both sides of reduce-bench identically reddens the cost test; removing the case-count guard reddens two, including the fail-closed one. Green after restoring. Verified: fmt, clippy -D warnings, 1114 workspace tests, schemas, ts-sdk, sbom, deny. Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

codeitlikemiley · 2026-08-24T00:09:35Z

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is exactly what M19.1 was written against. The models are still absent and that does not change; what changes is that when one arrives, there is something to judge it with.

No fabricated scores. The gates ship, not results.

M19.3 — `Score::beats(&incumbent, 10.0)`

The milestone's bar is relative ("beats heuristic by ≥10pt"); the existing meets_the_gate is an absolute floor answering a different question.

The margin is percentage points, not a fraction. Passing 0.10 and meaning ten points is the obvious way to get this wrong, both readings compile and both look right, so a test pins the unit.
A confidently-wrong challenger is refused regardless of margin. The dangerous quadrant is what route-bench exists to measure. A caller checking only the margin would ship a model that is better on average and catastrophic on the cases where the router acts on the answer.

M19.5 — `Report::cost_saving` / `cheaper_than(0.25, …)`

reduce-bench had no cost dimension at all. It scored retention and token ratio, which are not money — so "≥25% cheaper" was unmeasurable even with a model in hand.

Cost rather than tokens, because the two move independently: a reducer that keeps 40% of the tokens and hands them to a model at 3× the price is more expensive, and the old ratio-only scorecard would have called that a 60% win.

Prices are per token and supplied by the caller — panday-harness has no price table and must not grow one, or the gate starts drifting with a catalog. A saving against an incumbent that spends nothing returns None rather than a number. Both halves required: cheap-with-regressions is a reducer that saves money by losing the answer.

M19.7 — `Scorecard::beats(&base)`

On the artifact type every suite already emits, so it isn't agent-bench's alone.

It returns Option<bool>, and the None is the substance. Comparing a tuned run over 10 tasks against a base run over 41 is how this measurement gets faked without anyone lying: both numbers are real, the ratio is meaningless, and a plain bool would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. .unwrap_or(false) — what a caller will actually write — fails closed.

Falsified

Injected defect	Result
M19.3 margin treated as a fraction	red — unit test
M19.5 prices both sides identically	red — cost test
M19.7 case-count guard removed	red — two tests, incl. fail-closed

Green after restoring: 34/34 router, 42/42 harness, 13/13 types.

Numbering

No new milestone numbers — these are the gates of M19.3/5/7. Each bullet in docs/19 now records that its gate is executable while its model is not.

Verification

fmt · clippy --workspace --all-targets -D warnings · 1114 workspace tests, 0 failed · schemas · ts-sdk · sbom (609 components) · deny.

https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is precisely what M19.1 was written against. The models are still absent and that is unchanged; what changes is that when one arrives there is something to judge it with. **M19.3 — `Score::beats(&incumbent, 10.0)`.** The milestone's bar is relative ("beats heuristic by ≥10pt"); `meets_the_gate` is an absolute floor and answers a different question. The margin is percentage points, not a fraction: passing `0.10` and meaning ten points is the obvious way to get this wrong, both readings compile, so a test pins the unit. A confidently-wrong challenger is refused regardless of margin — the dangerous quadrant is what route-bench exists to measure, and a caller checking only the margin would ship a model that is better on average and catastrophic where the router acts on it. **M19.5 — `Report::cost_saving` and `cheaper_than(0.25, ..)`.** reduce-bench had **no cost dimension at all**: it scored retention and token ratio, which are not money, so "≥25% cheaper" was unmeasurable with a model in hand. Cost rather than tokens because the two move independently — a reducer keeping 40% of the tokens and handing them to a model at 3x the price is *more* expensive, and the old scorecard would have called that a 60% win. Prices are per token and supplied by the caller; this crate has no price table and must not grow one, or the gate drifts with a catalog. A saving against an incumbent that spends nothing is `None`, not a number. Both halves are required: cheap-with-regressions is a reducer that saves money by losing the answer. **M19.7 — `Scorecard::beats(&base)`**, on the artifact type every suite emits. It returns `Option<bool>` and the `None` is the substance: comparing a tuned run over ten tasks against a base over forty-one is how this gets faked without anyone lying — both numbers real, ratio meaningless — and a plain `bool` would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. `.unwrap_or(false)` fails closed. No new milestone numbers: these are the gates *of* M19.3/5/7, so each bullet records that its gate is now executable while its model is not. All three falsified. Making the margin a fraction reddens the unit test; pricing both sides of reduce-bench identically reddens the cost test; removing the case-count guard reddens two, including the fail-closed one. Green after restoring. Verified: fmt, clippy -D warnings, 1114 workspace tests, schemas, ts-sdk, sbom, deny. Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

codeitlikemiley · 2026-08-24T00:09:35Z

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is exactly what M19.1 was written against. The models are still absent and that does not change; what changes is that when one arrives, there is something to judge it with.

No fabricated scores. The gates ship, not results.

M19.3 — `Score::beats(&incumbent, 10.0)`

The milestone's bar is relative ("beats heuristic by ≥10pt"); the existing meets_the_gate is an absolute floor answering a different question.

The margin is percentage points, not a fraction. Passing 0.10 and meaning ten points is the obvious way to get this wrong, both readings compile and both look right, so a test pins the unit.
A confidently-wrong challenger is refused regardless of margin. The dangerous quadrant is what route-bench exists to measure. A caller checking only the margin would ship a model that is better on average and catastrophic on the cases where the router acts on the answer.

M19.5 — `Report::cost_saving` / `cheaper_than(0.25, …)`

reduce-bench had no cost dimension at all. It scored retention and token ratio, which are not money — so "≥25% cheaper" was unmeasurable even with a model in hand.

Cost rather than tokens, because the two move independently: a reducer that keeps 40% of the tokens and hands them to a model at 3× the price is more expensive, and the old ratio-only scorecard would have called that a 60% win.

Prices are per token and supplied by the caller — panday-harness has no price table and must not grow one, or the gate starts drifting with a catalog. A saving against an incumbent that spends nothing returns None rather than a number. Both halves required: cheap-with-regressions is a reducer that saves money by losing the answer.

M19.7 — `Scorecard::beats(&base)`

On the artifact type every suite already emits, so it isn't agent-bench's alone.

It returns Option<bool>, and the None is the substance. Comparing a tuned run over 10 tasks against a base run over 41 is how this measurement gets faked without anyone lying: both numbers are real, the ratio is meaningless, and a plain bool would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. .unwrap_or(false) — what a caller will actually write — fails closed.

Falsified

Injected defect	Result
M19.3 margin treated as a fraction	red — unit test
M19.5 prices both sides identically	red — cost test
M19.7 case-count guard removed	red — two tests, incl. fail-closed

Green after restoring: 34/34 router, 42/42 harness, 13/13 types.

Numbering

No new milestone numbers — these are the gates of M19.3/5/7. Each bullet in docs/19 now records that its gate is executable while its model is not.

Verification

fmt · clippy --workspace --all-targets -D warnings · 1114 workspace tests, 0 failed · schemas · ts-sdk · sbom (609 components) · deny.

https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

docs/19 states three acceptance gates in prose. None existed as code, so none could be failed — which is precisely what M19.1 was written against. The models are still absent and that is unchanged; what changes is that when one arrives there is something to judge it with. **M19.3 — `Score::beats(&incumbent, 10.0)`.** The milestone's bar is relative ("beats heuristic by ≥10pt"); `meets_the_gate` is an absolute floor and answers a different question. The margin is percentage points, not a fraction: passing `0.10` and meaning ten points is the obvious way to get this wrong, both readings compile, so a test pins the unit. A confidently-wrong challenger is refused regardless of margin — the dangerous quadrant is what route-bench exists to measure, and a caller checking only the margin would ship a model that is better on average and catastrophic where the router acts on it. **M19.5 — `Report::cost_saving` and `cheaper_than(0.25, ..)`.** reduce-bench had **no cost dimension at all**: it scored retention and token ratio, which are not money, so "≥25% cheaper" was unmeasurable with a model in hand. Cost rather than tokens because the two move independently — a reducer keeping 40% of the tokens and handing them to a model at 3x the price is *more* expensive, and the old scorecard would have called that a 60% win. Prices are per token and supplied by the caller; this crate has no price table and must not grow one, or the gate drifts with a catalog. A saving against an incumbent that spends nothing is `None`, not a number. Both halves are required: cheap-with-regressions is a reducer that saves money by losing the answer. **M19.7 — `Scorecard::beats(&base)`**, on the artifact type every suite emits. It returns `Option<bool>` and the `None` is the substance: comparing a tuned run over ten tasks against a base over forty-one is how this gets faked without anyone lying — both numbers real, ratio meaningless — and a plain `bool` would have hidden it. Refused on suite mismatch, on an empty run, and on differing case counts. `.unwrap_or(false)` fails closed. No new milestone numbers: these are the gates *of* M19.3/5/7, so each bullet records that its gate is now executable while its model is not. All three falsified. Making the margin a fraction reddens the unit test; pricing both sides of reduce-bench identically reddens the cost test; removing the case-count guard reddens two, including the fail-closed one. Green after restoring. Verified: fmt, clippy -D warnings, 1114 workspace tests, schemas, ts-sdk, sbom, deny. Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

codeitlikemiley marked this pull request as ready for review August 24, 2026 00:15

codeitlikemiley merged commit 124029f into mainAug 24, 2026
6 checks passed

codeitlikemiley deleted the m19-executable-gates branch August 24, 2026 00:15