tooling(qa): roll-up script over qa-run records — area × latest-verdict × staleness matrix for the wave anchor

[os-zhuang]

Maintainer authorization (2026-08-18, PM chat, skills-seat session session_01Rn7aaamsR99FXRqLcpL99q): 「同意你的落地动作」, approving the QA management-loop design, item 4.

What

A small script (suggested home scripts/qa/qa-rollup.mjs) that answers the one question no current view answers: "what is the latest verdict for every checklist item/area, and is it stale?" — today that takes reading 14 issues by hand.

• Input: qa-run issues (open and closed), read via the GitHub API — title convention QA run · <selector> · <judged>/<total> · <sha8> · <date> · <counts> plus the per-item verdict tables the records carry.
• Output: a markdown matrix — area × latest verdict counts (PASS/PARTIAL/FAIL/BLOCKED/NOT-RUN) × the subject sha it was measured on, flagged stale when that sha is behind the current comparison target; plus the open-record (close-out debt) count. Suitable for pasting as a wave-anchor comment.
• Latest-per-selector wins: a superseded record never shadows its replacement.
• It is a generated VIEW, not a tracker (one board rule): no state is written anywhere; the run records stay the only source of truth.

Readers (named, per the label/reader rule): the wave PM planning the next round's selectors; the maintainer reading test status.

Triage note: routing suggestion domain:devx (general scripts/ tooling) or domain:skills (PM-loop adjacent) — triage decides; not scripts/pm/**, so no fable path mandate.

[os-support-ai]

Triage: routed domain:devx — a roll-up script over qa-run records for the wave anchor is repo tooling (scripts/ / docs/qa machinery), not runtime surface. If it lands inside packages/qa instead, flag back rather than relabeling. (Triage seat, session session_019gCKd9EZfQ6MbGTnMHHJvW.)

---

Generated by Claude Code

[os-steve]

Claim: os-dev implementer, session session_01XqDQYVU5smx29ts9pAErja, branch claude/issue-9486-qa-rollup (pushed empty as a landing marker before the first edit).

Scope as dispatched: scripts/qa/qa-rollup.mjs — a generated view, no state written anywhere. Will measure the real qa-run corpus rather than assume the title convention holds, and print what does not parse.

---

Generated by Claude Code

[claude]

{
 "issue": 9486,
 "status": "done",
 "branch": "claude/issue-9486-qa-rollup",
 "pr": "https://github.com/objectstack-ai/objectstack/pull/9616",
 "premise_still_valid": true,
 "summary": "Added scripts/qa/qa-rollup.mjs: a generated view (no state written anywhere -- verified, git status is clean after a full run) that prints selector x latest-verdict x staleness over the qa-run records, plus close-out debt, superseded records, checklist areas with no record, and everything it could not parse. The card's premise holds and is delivered, but all three of its mechanical assumptions were falsified by measurement and each changed the design. (1) The title convention is followed by NOTHING: the documented form in .claude/skills/checklist-test/SKILL.md and the form described in the card are both 0/23; 23/23 records use a fifth shape with judged/total folded into a parenthetical on the selector in five distinct phrasings, absent entirely on one. (2) The per-item verdict tables do not have one shape and are deliberately NOT parsed: 9 of 23 records carry no markdown table at all, the other 14 carry 26 distinct header shapes, and most are clause-level rather than item-level -- they use the word 'verdict' for a different unit, so counting them would yield a confidently wrong matrix. The script reads the title counts (the only uniform machine-readable field, authored by the runner) and states in every run that it did not read the bodies. (3) Staleness had to become three-valued: this container is a shallow clone (boundary 713ccbc95, 2026-08-16) and the 92f26f75 wave predates it, so merge-base --is-ancestor answers false for commits that are almost certainly ancestors -- a boolean would have rendered the NINE STALEST RECORDS AS FRESH. 'Could not tell' renders as ? with its reason and is never counted as fresh. Corpus is 23 records, not the 14 the card assumed. No root package.json entry: scope discipline, and the file is held by two open PRs (see open_questions for the factual conflict finding).",
 "tests": "All run after the final commit, green at 506c30962 (the pushed head). `node scripts/qa/qa-rollup.mjs --self-test` -> '✓ qa-rollup --self-test: 60 assertions over 7 real-corpus title shapes' (fixtures are the real measured title shapes, no network). `node scripts/check-nul-bytes.mjs` -> 'OK (scanned 6172 text file(s) ... no raw ASCII control bytes)'; plus a manual grep -naP over the new file: clean. `npx eslint scripts/qa/qa-rollup.mjs --no-inline-config` -> exit 0 (run under flock /tmp/os-heavy-verify.lock). `node scripts/pm/dispatch-gates.mjs scripts/qa/qa-rollup.mjs` -> no path-derived gate family (0 matched of 105 discovered). The self-test caught two real defects during development, both fixed: a date validator that accepted 2026-13-99 (the date is the latest-wins sort key, so an impossible date had to become unparseable rather than silently order the roll-up), and a placeholder GH_TOKEN being forwarded as a Bearer header, which produced a 401 that reads as 'no access' when access was fine. LIVE RUN against the real corpus: 23 records, 23 parsed, 0 unparseable, 0 open, 1 REST call. REVERSE-VERIFICATION on real data: re-running with --target e4e5c6e3 flips the 14 rows measured on that sha from 'STALE (122 behind)' to 'current' while the 5 shallow-unknown rows correctly stay '?', proving the freshness column reads live git ancestry rather than a fixed label. Rate limit measured directly, answering the PM's untested assumption: REST core 1 call for the whole corpus against 15000/hr (14967 remaining); the script deliberately avoids the search API and GraphQL -- and the search quota was in fact exhausted mid-task by an unrelated MCP call, which is exactly the failure the design avoids.",
 "open_questions": [
 {
 "question": "The QA-run title convention documented at .claude/skills/checklist-test/SKILL.md:126 matches 0 of 23 real records, and that file is a governed human-merge-only surface I must not edit. Filed as #9617. Which direction should it take?",
 "options": [
 "A: update SKILL.md to document what runs actually write -- cheap, keeps all 23 existing records valid, but enshrines five phrasings of one field as the permanent contract",
 "B: tighten the convention to one shape and have new runs emit it -- contract-first, lets a parser be strict instead of tolerant, costs a migration call for the 23 existing records (cleanest: new records only)"
 ],
 "recommendation": "B, with the roll-up's 'Not parsed' section as the enforcement surface. It already prints any record it cannot read, so a tightened convention becomes visible the first time someone deviates rather than silently degrading. A alone preserves the tolerant-consumer shape, which is where authoring errors hide -- and this corpus already shows the cost: a record titled '(FULL area)' that judged 13 of its area's 33 items (#9351) reads as complete coverage from its title alone."
 },
 {
 "question": "Should --self-test be wired into a check:* gate family? I deliberately did NOT add a root package.json entry (scope: the card asks for a view, not a gate). The dispatch asked me to check the conflict factually, so: #9598 adds one line to the same scripts block and #9466 bumps devDependencies there. Notably #9598 introduces a single-writer-path guard, but its declared list contains ONLY .objectui-sha -- root package.json is not declared, so that gate would NOT have blocked this PR. The decision was scope discipline, not the guard.",
 "options": [
 "A: leave it as an on-demand script; --self-test is runnable directly and nothing rots silently because the parser has no other consumer yet",
 "B: add a check:qa-rollup-selftest entry once #9598 and #9466 land, so the parser is protected when the title convention next drifts"
 ],
 "recommendation": "B, but as a follow-up rather than in this PR. The parser now encodes measured facts about a corpus that demonstrably drifts, so it will rot -- but adding the entry today buys a needless merge conflict in a contended block for a gate that guards a script with no other caller yet."
 }
 ],
 "out_of_scope_findings": [
 "filed as #9617: the documented QA-run title convention in .claude/skills/checklist-test/SKILL.md matches 0 of 23 real records (governed surface -- reported, not edited)",
 "NOT filed, surfaced by the tool itself: checklist area `dashboards` (10 items) has never been run by any qa-run record -- this is precisely what the roll-up exists to show the wave PM, so filing an issue would duplicate the tool's purpose",
 "NOT filed, reported in #9617 as supporting evidence: #9351 records-forms is titled '(FULL area)' but judged 13 of the area's 33 items; the roll-up now renders it 13/33* with the asterisk marking a checklist-sourced denominator"
 ]
}

---

Generated by Claude Code