Skip to content

test(lint): pay the startup-verdict corpus sweep once, under a stated hook budget - #10911

Merged
os-zhuang merged 1 commit into
mainfrom
claude/issue-10838-corpus-sweep-budget
Aug 22, 2026
Merged

test(lint): pay the startup-verdict corpus sweep once, under a stated hook budget#10911
os-zhuang merged 1 commit into
mainfrom
claude/issue-10838-corpus-sweep-budget

Conversation

@claude

@claudeclaudeBot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Fixes#10838

What was removed vs what was widened

Removed: one of the two full corpus sweeps. The read + scan of every non-test .ts under packages/ (1,880 files / 28.15 MB as of this branch) ran once inside each of the two corpus cases, sharing nothing. It is now paid once in a beforeAll and both cases read the same findings.

Widened: nothing per-test. The sweep's budget moved from vitest's implicit default 5000ms per-test timeout to an explicit, commented 60s hook timeout.

Measurements I took on this branch

Corpus, re-derived (the card's numbers were from 12:23Z):

card, 12:23Zthis branch, ~16:55Z
files1,8721,880
bytes28.1 MB28.15 MB

The corpus grew by 8 files in about four hours. That is the trend the card is about, and it is why the fix is not a bigger number.

Sweep cost, isolated (tsx, this worktree):

corpus files=1880
sweep#1 (cold — includes lazy typescript load): 986ms findings=0
sweep#2 (warm): 465ms findings=0
sweep#3 (warm): 440ms findings=0

The test file, before and after (vitest --reporter=verbose, same machine):

casebeforeafter
reads a non-empty corpus2ms1ms
no package records a verdict the boot can still contradict972ms1ms
no ledger entry is stale467ms0ms
the sweep can still fire2ms2ms
file tests total1441ms816ms

So: 43% of the work is gone, and the largest single thing measured against the 5000ms per-test budget drops from 972ms to 2ms.

Why the timeout is still needed, and why it is 60s

Be honest about which half does the work. Sharing removes one warm sweep (~465ms locally). The ejecting run's 9144ms was the cold leg — the lazy ~9 MB typescript load plus a cold parse of the corpus on a shard that reported import 106.30s. That leg still has to happen once, and it is ~9.2x its local cost on a loaded queue shard.

Project the ejecting run onto the new shape: the hook does what its case 1 did, ~9144ms, and clears vitest's default hook timeout of 10000ms by 856ms. That is still a near-threshold budget on a shard whose load is the variable — which is exactly the failure mode this card is about. So the number is stated rather than defaulted.

60s is ~6.5x the worst wall-clock this sweep has ever been observed to take and ~60x the local one. The comment in the file says plainly that this is a liveness backstop — a wedged sweep must not pin a worker forever — and explicitly not a performance tripwire, because a snug budget over work that grows with the repo is guaranteed to red on somebody else's PR eventually. The cost stays legible without a tripwire: it is one hook now, and hook time lands in the file's own tests total.

The gate's meaning is unchanged, and that is proven, not asserted

Both of the file's original false-green refusals are untouched (reads a non-empty corpus; the sweep can still fire). Sharing introduces a third hazard, so the file now refuses that too:

  • Sharing is only sound because neither case mutates what the other reads — both derive (filter, map). A comment cannot hold that open, so the findings array and every finding in it are frozen.
  • The shared value starts undefined, not [], behind a corpusFindings() accessor that throws. A case reading a sweep that never ran would otherwise print as a clean audit over nothing — false green 1's shape, one level up.

Ablation — predictions written down before any leg ran

No build/dist step is involved: the test imports ./lint-startup-registry-verdict.js, which vitest resolves to the sibling .ts in src, and the corpus is read from the filesystem at run time. Each leg was mutate → prove the mutation on disk by marker count → run → restore → prove absent on disk.

legmutationpredictedobserved
Aplant a start() drawing a verdict from getRegisteredNodeTypes() and warning "will fail at execution time" (the #4771 shape) into the corpusno package records a verdict… reds alone× no package records a verdict…, both findings emitted with paths/lines; ✓ no ledger entry is stale; 1 failed / 3 passed
Bone LEDGER key no live finding producesno ledger entry is stale reds alone× no ledger entry is stale — the site is fixed, delete the line: packages/ablation/src/gone.ts:1::startup-open-vocabulary-verdict; ✓ no package records a verdict…; 1 failed / 3 passed
Cdrop the sweepResult = … publish in beforeAll (sweep still runs, result never published)both corpus cases red loudly, not silently green✅ both: Error: the corpus sweep did not run — this case would otherwise report a clean audit over a corpus it never swept (#10838); 2 failed / 2 passed
Dwith leg A's plant present, mutate findings[0].message in the first casethat case reds on the frozen objectTypeError: Cannot assign to read only property 'message' (vitest names the frozen object after it)

A and B are the load-bearing pair: each case reds individually off the shared array, so neither has quietly stopped testing anything. C proves the new refusal is not decorative. D is worth noting for a reviewer: TypeScript did not catch the mutation statically — corpusFindings() returns readonly Finding[], which freezes the array shape but not the elements' properties — so the runtime freeze is the thing that actually holds the invariant.

Restore was proven byte-identical by regenerating the file deterministically from the pristine HEAD copy and comparing hashes:

S1 (post-ablation restore) = 5529ce217d459212b811db26fbec78e9395460b0dc3ce94b1228224e7d910db9
S2 (regenerated from HEAD) = 5529ce217d459212b811db26fbec78e9395460b0dc3ce94b1228224e7d910db9
BYTE-IDENTICAL: yes

git status --porcelain showed the one intended file and no untracked ablation residue.

Verification

Final green, at the commit this PR carries (b97a70c9a7):

✓ … > reads a non-empty corpus 1ms
✓ … > no package records a verdict the boot can still contradict 1ms
✓ … > no ledger entry is stale 0ms
✓ … > the sweep can still fire (#4690 …) 2ms
Test Files 1 passed (1) Tests 4 passed (4)
Duration 1.37s (… tests 854ms …)

Whole package: pnpm --filter @objectstack/lint testTest Files 80 passed (80), Tests 2246 passed | 5 skipped (2251), exit 0. pnpm --filter @objectstack/lint typecheck → exit 0, no error TS. Both filters echoed @objectstack/lint@17.1.0, and --filter @objectstack/definitely-not-a-package was run once as a negative control: it printed No projects matched the filters and exited 0 (#10853), so the reading above is confirmed rather than assumed.

Gate union re-derived at b97a70c9a7 with node scripts/pm/dispatch-gates.mjs (no hand-built path list), all green, each quoting the gate's own verdict line:

  • check:cross-package-test-inputsOK: 13 package(s) read outside themselves, all declared… (+ 104 self-test cases)
  • check:slot-lookup✓ slot-lookup ratchet holds: 107 unswept site(s)… none new
  • check:test-source-aliasOK — 72 packages with tests scanned…
  • check:type-source-resolutionOK — 76 packages with a tsconfig.json scanned…
  • check-ci-filter-parity.mjsOK: all 82 declared cross-package glob(s)…
  • check-plugin-teardown-shape.mjs✓ … 57 Plugin implementation(s) across 4408 source(s)
  • check-affected-docs.mjs✓ affected-docs self-test: 323 cases pass.
  • check:query-options-erasure✓ ratchet holds… baseline key set verified against 409077e: no files added
  • check:type-check-coverageOK — 64/77 workspace packages type-checked…
  • check:engine-double-contractOK — 376 pinned, 133 in the DEBT ledger, 2 exempt
  • check:where-matcher✓ conformance holds: 275 matcher(s)… none new
  • check-nul-bytes.mjsOK (scanned 6301 text file(s)… no raw ASCII control bytes)

One declared narrowing:check:type-check-debt --re-measure was not run locally. It refuses to measure on an unbuilt worktree and needs turbo run build across the whole workspace, which would hold this container's shared verify lock for a long time while three other agents are queued on it. The reason it is convention-triggered here is "a new test file that does not typecheck cleanly" — this PR adds no new test file, edits one that already typechecks, and pnpm --filter @objectstack/lint typecheck is green at this commit. CI runs it regardless.

Release declaration

Test-only: the change is confined to packages/lint/src/lint-startup-registry-verdict.corpus.test.ts, which tsup never builds into dist and package.json#files never publishes. Nothing releases, so this PR carries skip-changeset rather than a changeset — derived from pr-automation.yml's changeset-check (its two exemptions both mean "this PR declares no release of its own"), not assumed.


Generated by Claude Code

… hook budget
The corpus read + scan (1880 files / 28.15 MB) ran twice, once inside each of
the two corpus cases, sharing nothing — and both cases sat under vitest's
default 5000ms per-test timeout. On a loaded merge-queue shard the first case
measured 9144ms and ejected PR #10733, which never touched packages/lint, from
the queue; every entry behind it rebuilt.
Hoist the sweep into one beforeAll and give it an explicit, commented budget:
- Work removed: file `tests` total 1441ms -> 816ms locally; the largest thing
measured against the per-test budget drops from 972ms to 2ms.
- The budget is a hook timeout, deliberately loose (60s ~= 6.5x the worst
wall-clock ever observed for this sweep). It is a liveness backstop, not a
performance tripwire — a snug budget is what ejected an unrelated PR, and the
work grows with the repo (1872 files at 12:23Z, 1880 four hours later) while
the wall-clock varies with shard load.
Sharing is only sound because neither case mutates what the other reads, so the
findings array and every finding in it are frozen, and the shared value starts
`undefined` rather than `[]` behind a `corpusFindings()` accessor that throws.
That last part is a third false green the file now refuses: a case reading a
sweep that never ran would otherwise print as a clean audit over nothing.
Both gate cases still fire individually — proven by ablation, see the PR body.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DdCnBGcHeufjrq7drTD3wt
@claudeclaudeBot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 32550265418 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

失败的 job(日志抽取,best effort):

  • Console Pin Gate — 失败步骤: Build the Console SPA at the pinned objectui SHA

    ✗ Build failed in 5.99s
    

↳ 失败原因 是判读的关键:超时Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

跨 PR 相同签名(24h,按失败测试文件聚合):

  • ⚠️本次没有可用的聚合签名(日志里没有能解析出测试文件名的 FAIL 行)—— 这不是「没有同签名的其他 PR」,是这一轮没测到。跨 PR 聚合本次不可用,请手工比对其他 PR 的同类评论。
  • ⚠️ 24h 评论账本没读完(超过 5 页仍未读到窗口尽头),所以上面的「不同 PR 数」是下界,不是全量。

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 70 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/sskip-changesetPR has no user-facing published change; bypasses the changeset gatetests

Projects

None yet

2 participants

@os-zhuang@claude