test: re-baseline the evaluation benchmark - #203
Merged
Conversation
benchmark/run.ts has been failing on clean master since the baselines were last regenerated in Phase 12. Everything from Phases 13–29 was unaccounted for, so the guard has been dark for a long stretch — including the whole of Phase 29. Every delta traces to a deliberate change, checked one at a time rather than accepted in bulk: new category: usability Phase 27 slice C new category: coverage Phase 24 slice C APCA advisories (8 across the Phase 25 (info-only, never gates) corpus) "Display text at 28px with Phase 25 tracking advisory default tracking" (4) radius census 5 -> 4 distinct Phase 27 slice B excluded pill radii; the hero's cornerRadius: 999 pill stopped counting eyebrow census 4 -> 3 Phase 28 slice C's detectTable three-row rule now finds a table in the hero, whose header is excluded from the census by design (the same detection produces the two new coverage findings) spacing message reworded Phase 29 slice D — variety is measured against the DECLARED scale, and corpus canvases declare none, so they take the no-scale branch The corpus keeps its discriminating power: 98 / 82 / 72 across minimal, bad-contrast and hero. WORTH KNOWING, and not a bug: bad-contrast rose 77 -> 82 without improving. Its color score is still 0. The rise is dilution — two new categories scoring 100 lifted the weighted average while the fixture's actual defect was untouched. Adding categories makes every screen that does not exercise them score higher, so scores are not comparable across phases that changed the category set.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
npx tsx benchmark/run.tshas been failing on clean master since the baselines were last regenerated in Phase 12. Everything from Phases 13 through 29 was unaccounted for, so this guard has been dark for a long stretch — including the whole of Phase 29, where I hit it repeatedly and each time had to re-verify it was pre-existing rather than mine.Every delta, attributed
The point of a baseline is to catch unintended evaluator drift, so each changed number had to be explainable by a known change rather than accepted in bulk:
usabilitycoveragecornerRadius: 999pill stopped countingdetectTablethree-row rule now finds a table in the hero, and a table's header is excluded from the census by design — the same detection produces the two new coverage findingsTwo of these took actual digging. The eyebrow count dropping by exactly one looked arbitrary until the coverage findings gave it away: the same table detection explains both. I checked it rather than waving at "Phase 28 changed the tells".
The corpus still discriminates
98 / 82 / 72 across
minimal,bad-contrastandhero— the spread that makes the benchmark useful is intact.One finding worth more than the re-baseline
bad-contrastrose 77 → 82 without improving. Its color score is still 0:The rise is dilution. Two new categories scoring 100 lifted the weighted average while the fixture's real problem was untouched.
That generalises: adding an evaluator category raises the score of every screen that doesn't exercise it. So scores are not comparable across phases that changed the category set, and the
> 95bar has quietly become easier to reach as categories accumulated — eight now, against five when that bar was set. Not a bug, and not something to fix in this PR, but it is the kind of thing a dark benchmark hides. Flagging it rather than burying it in a baselines diff.Verification
benchmark/run.tspasses.test-evaluate20,test-discoverability117/117 — the change is data-only, but those are the two that would notice.