Skip to content

test: re-baseline the evaluation benchmark - #203

Merged
vicmaster merged 1 commit into
masterfrom
benchmark-rebaseline
Aug 17, 2026
Merged

test: re-baseline the evaluation benchmark#203
vicmaster merged 1 commit into
masterfrom
benchmark-rebaseline

Conversation

@vicmaster

Copy link
Copy Markdown
Owner

npx tsx benchmark/run.ts has been failing on clean master since the baselines were last regenerated in Phase 12. Everything from Phases 13 through 29 was unaccounted for, so this guard has been dark for a long stretch — including the whole of Phase 29, where I hit it repeatedly and each time had to re-verify it was pre-existing rather than mine.

Every delta, attributed

The point of a baseline is to catch unintended evaluator drift, so each changed number had to be explainable by a known change rather than accepted in bulk:

deltacause
new category usabilityPhase 27 slice C
new category coveragePhase 24 slice C
8 APCA advisories across the corpusPhase 25 — info-only, never gates
4 × "Display text at 28px with default tracking"Phase 25 tracking advisory
radius census 5 → 4 distinctPhase 27 slice B excluded pill radii; the hero's cornerRadius: 999 pill stopped counting
eyebrow census 4 → 3Phase 28 slice C's detectTable three-row rule now finds a table in the hero, and a table's header is excluded from the census by design — the same detection produces the two new coverage findings
spacing message rewordedPhase 29 slice D — variety is measured against the declared scale, and corpus canvases declare none, so they take the no-scale branch

Two of these took actual digging. The eyebrow count dropping by exactly one looked arbitrary until the coverage findings gave it away: the same table detection explains both. I checked it rather than waving at "Phase 28 changed the tells".

The corpus still discriminates

98 / 82 / 72 across minimal, bad-contrast and hero — the spread that makes the benchmark useful is intact.

One finding worth more than the re-baseline

bad-contrast rose 77 → 82 without improving. Its color score is still 0:

 color 0 -> 0 ← the fixture's actual defect, unchanged
usability — -> 100 ← new category
coverage — -> 100 ← new category

The rise is dilution. Two new categories scoring 100 lifted the weighted average while the fixture's real problem was untouched.

That generalises: adding an evaluator category raises the score of every screen that doesn't exercise it. So scores are not comparable across phases that changed the category set, and the > 95 bar has quietly become easier to reach as categories accumulated — eight now, against five when that bar was set. Not a bug, and not something to fix in this PR, but it is the kind of thing a dark benchmark hides. Flagging it rather than burying it in a baselines diff.

Verification

benchmark/run.ts passes. test-evaluate 20, test-discoverability 117/117 — the change is data-only, but those are the two that would notice.

benchmark/run.ts has been failing on clean master since the baselines
were last regenerated in Phase 12. Everything from Phases 13–29 was
unaccounted for, so the guard has been dark for a long stretch —
including the whole of Phase 29.
Every delta traces to a deliberate change, checked one at a time rather
than accepted in bulk:
new category: usability Phase 27 slice C
new category: coverage Phase 24 slice C
APCA advisories (8 across the Phase 25 (info-only, never gates)
corpus)
"Display text at 28px with Phase 25 tracking advisory
default tracking" (4)
radius census 5 -> 4 distinct Phase 27 slice B excluded pill radii;
the hero's cornerRadius: 999 pill
stopped counting
eyebrow census 4 -> 3 Phase 28 slice C's detectTable
three-row rule now finds a table in
the hero, whose header is excluded
from the census by design (the same
detection produces the two new
coverage findings)
spacing message reworded Phase 29 slice D — variety is measured
against the DECLARED scale, and corpus
canvases declare none, so they take
the no-scale branch
The corpus keeps its discriminating power: 98 / 82 / 72 across minimal,
bad-contrast and hero.
WORTH KNOWING, and not a bug: bad-contrast rose 77 -> 82 without
improving. Its color score is still 0. The rise is dilution — two new
categories scoring 100 lifted the weighted average while the fixture's
actual defect was untouched. Adding categories makes every screen that
does not exercise them score higher, so scores are not comparable across
phases that changed the category set.
@vicmaster
vicmaster merged commit 8ff503a into masterAug 17, 2026
@vicmaster
vicmaster deleted the benchmark-rebaseline branch August 17, 2026 19:34
@vicmastervicmaster mentioned this pull request Sep 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vicmaster