Skip to content

check:platform-checklist validates oracle against a closed vocabulary but never reads traps — the drift that let 8 undocumented values in is still open #10647

Description

@os-zhuang

Filed unassigned from the #10416 execution seat. #10416 documents the eight traps values that had drifted in; this card is the other half of that finding — the reason they drifted in at all, which #10416's PR deliberately does not build.

The gap

scripts/check-platform-checklist.mjs enforces a closed vocabulary for every other enum-ish field on a checklist item:

constSTATUSES=newSet(['active','draft','retired']);constPRIORITIES=newSet(['P0','P1','P2']);constSURFACES=newSet(['browser','api','cli','build','mixed']);constORACLES=newSet(['api','network','screenshot','dom','log','test','build']);constBLOCKED_BY=newSet(['fixture','environment','dependency','product-bug']);

…and checks them (if (!ORACLES.has(c.oracle)) where(...), line 131). The string traps does not appear anywhere in the file. So an item can carry any traps value at all and the validator is green.

Measured on main at 6b0be02209, parsing all 15 docs/qa/platform-checklist/areas/*.json (205 items): 19 distinct traps in use, 11 documented in RUNNER.md's ### Trap vocabulary table, 8 used-but-undocumented.#10416's PR takes that to 19/19/0 — but nothing holds it there.

Why it matters beyond the eight

RUNNER.md rule 3 tells a runner to "check the traps field and rule each listed trap out". Two ways that instruction silently becomes unexecutable, and the validator sees neither:

  1. A new undocumented value, exactly as the eight arrived.
  2. A typo in a documented onehydration-races, wrong-persona — which is worse than case 1, because it reads as a documented trap right up until someone greps the table for it. hydration-race is on 79 of 205 items; a typo on one of them is a trap nobody rules out and nobody notices.

Both fail closed if the vocabulary is checked, and the oracle check is the in-file precedent for exactly that shape.

Why it was not built in #10416's PR

It is not a sixth Set. Every vocabulary above is hardcoded in the script; the trap vocabulary's source of truth is a markdown table in RUNNER.md, so the check has to either:

  • (a) parse the ### Trap vocabulary table out of RUNNER.md — a markdown-table parser with its own failure modes (heading renamed, table moved, a row's backtick spelling changed), which fails open if the parse silently finds zero rows unless it also asserts a non-empty vocabulary; or
  • (b) hardcode a TRAPS set in the script — which reintroduces the same drift one level up, between the script and RUNNER.md, with nothing catching it.

(a) is the right shape and wants the "zero hits needs a positive control" discipline the repo already applies elsewhere. That is a design decision, not a one-liner, which is why it is a card rather than a rider.

Suggested shape

In scripts/check-platform-checklist.mjs: read RUNNER.md, extract the trap-table row names, assert the extracted set is non-empty (a zero-row parse must be a loud failure, never an empty allow-list), then check every item's traps entries against it. Both directions are worth flagging, mirroring the coverage ratchet's style: a used-but-undocumented value, and a documented-but-unused row (0 today, after #10416).

Note check:platform-checklist is not CI-wired by maintainer decision (README "Operating cadence") — this adds a check to the manual-cadence validator, it does not change that posture.

Refs: scripts/check-platform-checklist.mjs (the ORACLES check at line 131 is the precedent) · docs/qa/platform-checklist/RUNNER.md (rule 3 + the trap table) · #10416 (the eight definitions; its first-touch grading proposed this as disposition B).


Generated by Claude Code

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions