Skip to content

fix(scripts): refuse a logless check-test-completeness run instead of answering exit 1 - #13152

Merged
os-elon merged 1 commit into
mainfrom
claude/issue-13110-test-completeness-refusal
Aug 29, 2026
Merged

fix(scripts): refuse a logless check-test-completeness run instead of answering exit 1#13152
os-elon merged 1 commit into
mainfrom
claude/issue-13110-test-completeness-refusal

Conversation

@os-elon

Copy link
Copy Markdown
Collaborator

Fixes#13110

A gate that cannot measure must not answer in the vocabulary of a verdict.

Run with no turbo-test-log, scripts/check-test-completeness.mjs exited 1 and
printed its usage banner. Exit 1 plus text on a gate script reads as a finding — and
the bare form is not a mistake anyone chose: the gate family
scripts/pm/dispatch-gates.mjs derives NAMES this script with no argument, so the
tooling hands out the invocation that cannot succeed. Six devs on six unrelated cards
each re-derived that it was an invocation error. All six got it right; that is the
point. The cost was six correct readers paying the same re-derivation, with a seventh,
less careful, one red PR away.

What changed

A missing or unreadable log now answers with the repo's own refusal convention —
PREREQUISITE NOT MET, exit 3, plus a line naming what the gate needs and stating
that nothing was measured. Copied from the established shape, not minted:
check-dual-build-cjs-loads.mjs (exit 3, "This is NOT a pass: nothing was measured"),
scripts/pm/check-half-states.mjs (EXIT_PREREQUISITE_NOT_MET = 3) and
scripts/import-prerequisite.mjs's frame.

The refusal text names the derivation explicitly, so the next reader who arrives from
the derived family is told on the spot that the local reading is NOT MEASURED.

The argv classification became a pure exported function (invocationVerdict), so the
exit-code contract is pinned by the self-test that already runs on every invocation
rather than asserted in a comment.

The deliberate boundary

Exit 3 is for a missing or unreadable log — an input the world did not supply. A
malformed command line (an unknown flag, half of the scheduled/package-list flag
pair) stays exit 1: that is a typo in the caller's own line, it names itself as one,
and nothing emits it — the derived family prints the bare form, never a misspelled
flag. Pinned in the self-test so it stays a decision and not a leftover.

Precondition verified, not assumed

Every reference to this script in the tree, and what it does with the exit code:

callerpasses the log argument?branches on the exit code?
.github/workflows/ci.yml Test Core (:666)yes, plus both optional flagsno — plain run:, any non-zero fails the step
.github/workflows/ci.yml Dogfood (:1166)yesno — same
scripts/check-self-test-wired.mjsdoes not invoke it; pins the literal selfTest({ quiet: true }) in its code, which is preservedn/a
check-agent-test-spelling.mjs, check-entry-guard.mjs, dispatch-gates.mjs, run-with-stall-guard.mjsprose mentions in their own headersn/a
packages/spec/src/shared/strict-object.test.tsa comment citing precedentn/a

Nothing keys on exit 1. Both CI call sites also guard the log's existence with a
[ ! -f ... ] test before invoking, so neither new branch is reachable there.

CI behaviour did not move — measured in both directions

Both CI invocation shapes were run against the unmodified script (restored from the
HEAD blob) and the modified one, and the output compared byte for byte:

invocationbeforeafteroutput
Test Core shape: log + scheduled + package-listexit 0exit 0byte-identical
Dogfood shape: log onlyexit 0exit 0byte-identical
log with a real shortfall (a finding)exit 1exit 1byte-identical
readable log, no summariesexit 0exit 0byte-identical
unknown flagexit 1exit 1byte-identical
half a flag pairexit 1exit 1byte-identical

And the changed direction, reproducing the control the card asked for — the
unmodified script and the changed one, both invoked bare:

invocationunmodifiedmodified
no argumentexit 1 + usage bannerexit 3 + PREREQUISITE NOT MET
path that does not existexit 1exit 3
path that is a directoryexit 1exit 3

The pin, ablated

The self-test case is the durable half, so it was proven able to fail. With the
implementation committed first, the refusal code was mutated back to the finding code;
the mutation was confirmed on disk by anchored counts (removed text 1 to 0, injected
text 0 to 1) and by a blob-hash change against the HEAD blob, never by an editor exit
code. The pin then red:

Error: bare invocation: not the refusal code: got 1, want 3

on both --self-test and an ordinary bare invocation, since the self-test runs on every
invocation. The restore leg was proven the same way: anchored counts back, worktree blob
hash equal to the HEAD blob, git diff HEAD empty. No build leg applies — node executes
this .mjs directly and nothing resolves it through a package's exports or dist.

Gates

Derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack
(not stale: derived from this branch's tree at the commit below). All run at
589e6644, each read from the gate's own printed verdict line:

check:agent-test-spelling · check:bash32-floor · check:cli-command-ids ·
check:cross-package-test-inputs · check:entry-guard · check:parse-guard ·
check:pnpm-filter-targets · check:watch-hint-literal · check-ci-filter-parity ·
check-cross-package-test-inputs · check-self-test-wired · check-shard-attestation
— all green.

Convention pair for a gate-script edit, both green:
scripts/pm/bare-root-worklist.mjs --self-test (40 recorded verdicts, none stale,
missing or contradicted) and check:pm-dispatch-gates (872 cases).

Plus check:nul-bytes, check:published-files, and the repo-wide
eslint . --no-inline-config5414 files, 0 errors, 0 warnings, run under the
shared verify lock (VERDICT command-exit 0).

The thirteenth derived family member is this script itself, invoked bare. It now reads
exit 3, PREREQUISITE NOT MET — which is the fix, observed through the very tool that
produced the complaint.

No changeset — derived, not asserted

Across all 80 tracked manifests, 69 publishable: zero files entries name any
scripts/ path and zero escape their own package (control: 69 entries name dist).
This path ships in nothing, so the PR releases nothing. skip-changeset applied.

Out of scope, deliberately

Remedy B — the derivation no longer emitting an invocation that cannot succeed — is
not in this PR. scripts/pm/dispatch-gates.mjs is held by #13129. Noted for
whoever picks it up: with A landed, the gate refuses legibly however it is invoked, but
the derived family still prints a command line nobody should run. A is worth having
either way; B remains a separate call once the file is free.


Generated by Claude Code

… reporting exit 1
A gate that cannot measure must not answer in the vocabulary of a verdict. Run
with no turbo-test-log, check-test-completeness exited 1 and printed its usage
banner -- indistinguishable at a glance from a finding, and the bare form is
exactly what the gate family `scripts/pm/dispatch-gates.mjs` derives prints. Six
devs on six unrelated cards each re-derived that it was an invocation error.
A missing or unreadable log now answers with the repo's own PREREQUISITE NOT MET
/ exit 3 refusal convention (check-dual-build-cjs-loads, check-half-states),
naming what it needs and saying that nothing was measured. A malformed command
line keeps exit 1: a typo in the caller's own line is not an absent prerequisite,
and nothing emits one.
CI behaviour is unchanged: both call sites in ci.yml pass the argument and guard
the file's existence first, and neither branches on the exit code. Verified by
running both CI shapes before and after -- byte-identical output, same codes.
The argv classification is now a pure exported function, so the exit-code
contract is pinned by the self-test that already runs on every invocation rather
than asserted in a comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPrUz21stTFhJRUirdc4yw
@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 33242331510 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

失败的 job(日志抽取,best effort):

  • Test Core (1/6) — 失败步骤: Run this shard's tests

    @objectstack/cli:test: FAIL test/serve-publishes-bound-port.e2e.test.ts > #13062 the non-zero half — nothing an ordinary boot publishes may move > follows the DEV AUTO-SHIFT onto the port it really ↳ 失败原因: @objectstack/cli:test: Error: ENOENT: no such file or directory, open '/tmp/os-bound-port-home-MjBVsg/runtime.env_local.json'
    

↳ 失败原因 是判读的关键:超时Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

跨 PR 相同签名(24h,按失败测试文件聚合):

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 10 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

Merged via the queue into main with commit de3eb88Aug 29, 2026
34 checks passed
@os-elon
os-elon deleted the claude/issue-13110-test-completeness-refusal branch August 29, 2026 08:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/mskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants

@os-elon@claude