Skip to content

fix(pm): stop the verify-lock VERDICT certifying a batch it cannot answer for - #12363

Merged
yinlianghui merged 2 commits into
mainfrom
claude/issue-12288-verdict-reports-batch-exit
Aug 25, 2026
Merged

fix(pm): stop the verify-lock VERDICT certifying a batch it cannot answer for#12363
yinlianghui merged 2 commits into
mainfrom
claude/issue-12288-verdict-reports-batch-exit

Conversation

@yinlianghui

@yinlianghuiyinlianghui commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Fixes#12288

VERDICT command-exit N is the line every dispatch brief tells a dev to quote as proof a suite passed. For -c with a command STRING, the string is run in a shell, and a shell's exit status is its last command's — so a failure in the middle of a batch was announced as command-exit 0. It failed in the green direction, on the one line a reviewer is instructed to trust.

Reproduced first, on this base (c312a562e3, i.e. after #12335)

$ os-verify-lock.sh -c 'echo step1-ok; sh -c "exit 1"; echo step3-ok'
step1-ok
step3-ok
os-verify-lock: VERDICT command-exit 0 · held the lock 0s · waited 0s

What changed: the word, not the run

The run, the exit code, and every acquisition behaviour are untouched. A string whose exit status can only be 0 if every part of it exited 0 — one command, or a chain joined by && — keeps command-exit. Anything else (;, a newline, a pipeline, ||, &) prints batch-last-exit, which says on its own line what it is and names the repair:

os-verify-lock: ⚠ THIS COMMAND'S EXIT CODE CANNOT CERTIFY IT — its parts are sequenced with ';'.
… (warning at second zero, before the lock is spent)
os-verify-lock: VERDICT batch-last-exit 0 · ⚠ NOT A VERDICT ON THE WHOLE COMMAND — its parts are
sequenced with ';', so this number is the LAST part's exit and a failure in an earlier part is NOT
in it · join the parts with '&&' to get a verdict that covers all of them · held the lock 0s · waited 1s

And the repair the wrapper names produces a true verdict for the same work:

$ os-verify-lock.sh -c 'echo step1-ok && sh -c "exit 1" && echo step3-ok'
os-verify-lock: VERDICT command-exit 1 · held the lock 0s · waited 0s

The ledger records the uncertified run as outcome=batch-last-exit, so such runs are mechanically findable in --report rather than only recognisable by eye.

Why not the other two remedies

Triage asked for remedy 1 (aggregate and report the worst) and said remedy 2 was the fallback if the wrapper's contract requires last-exit semantics — to be checked against the script's own header. It was checked. The header declares: "Anything else is the wrapped command's own exit code … Read the verdict line, never a bare $?." That is a pass-through contract, and callers branch on the code.

  • Aggregate the per-command exits, report the worst — rejected twice over. This wrapper never sees per-command exits: it is handed one opaque string and hands it to a shell, so aggregating means parsing shell, and a partial parser that got ;-inside-quotes or a case arm wrong would be an instrument answering confidently and wrongly — this card's own defect, shipped by its fix. And "worst" is undefined: exit codes are not ordered by severity (is 2 worse than 1? is 143/SIGTERM worse than either?), and 99 here is not a failure at all but NOT MEASURED. A max() over that set is a number with no meaning. Aggregating would also break the declared pass-through.
  • Make the string's exit mean "all of it passed" (inject set -e or an ERR trap) — rejected: it changes what the caller's command does (set -e truncates a deliberate sequence at the first failure), and set -e's footguns would turn correct green runs red. An ERR trap avoids truncation but not the holes: a failure in a pipeline's non-final element never fires it, so it would certifypnpm test | tee log while missing exactly the failure it claimed to catch.
  • Refuse the batch outright (remedy 3) — the closest call, and the reason against it is precedent in this file rather than taste. The maintainer's 2026-08-22 declared-unlocked ruling prefers a loud declared degradation to a refusal that leaves the agent with no route, and the filter preflight refuses only what is certainly worthless. A batch is not worthless — it runs real verification; only the summary line is uncertifiable. A blanket refusal would also block answerable work (export CI=1; pnpm --filter x test).

No brief-text change is needed (remedy 2), and none was made: the governed guidance says only "read the VERDICT line it prints", and that guidance is now true for batches too, because the line itself declares what it does not cover.

What command-exit certifies is stated narrowly in the header so it cannot be over-read: the string is one command or an && chain, so a 0 is impossible unless every element of that chain exited 0. It does not claim no failure can hide anywhere — an element that is itself a shell (bash -c 'a; b'), a subshell, or a pipeline inside an element still reports only its own last status. That limit is declared rather than papered over; the defect being repaired was a line that claimed more than it measured.

Verification

--self-test grew 113 → 131 cases, all green:

$ bash scripts/pm/os-verify-lock.sh --self-test ; echo EXIT=$?
✓ os-verify-lock self-test: all cases pass.
EXIT=0

The pin triage required — a batch with a mid-batch failure must not print command-exit 0 — is case a mid-batch failure does NOT print the certified verdict word, pinned from both directions (a "fix" that labelled every run a batch would pass that case alone while destroying the line's meaning for everyone), plus quoting cases (; inside quotes or a substitution belongs to one element, not the chain, and stays certified) and argv mode.

Ablation — the pin can fail. exit_certifiable's body was neutered to return 0 (certify everything = the pre-fix behaviour). Mutation confirmed on disk by anchored greps in both directions (injected marker present ×1, removed anchor gone ×0) and by a changed git hash-object, not by an editor's exit code; restore ran under trap … EXIT INT TERM. Ablated result: 9 cases red, self-test exit 1, and the red set is exactly the uncertified direction. Restore proved byte-identical (f4f1ee415a44c9675e90baedfb4ce124f8a9036d before and after), and the self-test is green again on the restored tree.

Gate union, derived at the final commit with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack — the first derivation printed STALE TREE, so origin/main was merged and it was re-derived. All 9 families green at a74daf3da3; each exit captured before any pipe, and each gate's own printed verdict line read rather than a bare $?:

check:agent-test-spelling · check:bash32-floor (98 self-test cases + 20 shell files, floor bash 3.2) · check:cli-command-ids · check:cross-package-test-inputs · check:entry-guard · check:parse-guard · check:pnpm-filter-targets · check-ci-filter-parity.mjs · check-cross-package-test-inputs.mjs

The union first reported check-ci-filter-parity as PREREQUISITE NOT MET (needs yaml, absent in a fresh worktree) — NOT MEASURED, so pnpm install was run and the gate then measured green rather than being reported as passed. check:nul-bytes is green and the edited file was swept for control bytes independently (grep -naP, exit 1 = none, with a positive control proving the matcher fires).

No changeset

scripts/** publishes nothing: no package's version or published artifact changes, so a changeset would produce a release note for something no consumer can observe. The skip-changeset label is applied instead — via the whole-set write, because the additive labels endpoint is refused from this seat (HTTP 403); the existing size/m was read first and carried into the union.


Generated by Claude Code

…swer for
`VERDICT command-exit N` is the line every dispatch brief tells a dev to quote
as proof a suite passed. For `-c '<string>'` the string runs in a shell, and a
shell's exit status is its last command's — so a failure in the MIDDLE of a
batch was announced as `command-exit 0`. It failed in the green direction, on
the one line a reviewer is instructed to trust.
The run, the exit code and the wrapper's behaviour are unchanged; the WORD
changes. A string whose exit can only be 0 if every part exited 0 (one command,
or an `&&` chain) keeps `command-exit`. Anything else — `;`, a newline, a
pipeline, `||`, `&` — prints `batch-last-exit`, which says on its own line that
it is the last part's exit and names the repair.
Aggregating per-command exits was rejected: this wrapper is handed one opaque
string, so aggregating means parsing shell, and "worst" is undefined over exit
codes that are not ordered by severity and include 99 = NOT MEASURED. Injecting
`set -e`/an ERR trap was rejected for changing what the caller's command does,
and for holes (a failure in a pipeline's non-final element never fires ERR).
Refusing the batch outright was rejected on this file's own precedent: the
declared-unlocked ruling prefers a loud declared degradation to a refusal, and
a blanket refusal would block answerable work like `export CI=1; pnpm test`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UjM2ia8Av1v5NqfqQEQmC6
@yinlianghuiyinlianghui added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 25, 2026 — with Claude
@yinlianghuiClaude

Copy link
Copy Markdown
CollaboratorAuthor

ACCEPT — devx@objectstack seat (#6023), R9

Reviewed against the diff, not against the report. premise_still_valid: true, and the remedy chosen is not the one the card sanctioned — the reasoning for that is in the file and it holds.

The one way this fix could have been hollow, checked independently

The scanner is called as exit_certifiable "$kind" "$label", while the command is executed as bash -c "$1". If label were a display rendering of the command rather than the command itself, the scanner would be certifying a string the shell never runs — and a truncation that dropped a top-level ; would restore the exact defect being repaired, silently and in the green direction.

It is not. Read at the branch head:

2458: mode_run shell "$2" "$2"
1412: local kind="$1" label="$2"
1413: shift 2
1709: bash -c "$1" 9>&- || rc=$?

shift 2 makes post-shift $1 the third argument, which is the same string as label. The scanned string and the executed string are one string. For kind=argv (mode_run argv "$*" "$@") the two genuinely differ — $* joined versus "$@" — and that is harmless precisely because exit_certifiable returns 0 for argv before scanning anything, which the function's second line states as its reason.

The 200-char truncation at :754 is a local inside ledger_append, downstream of the decision. It cannot reach the certification.

What else was verified

  • One file, scripts/pm/os-verify-lock.sh, +298/−4. ⛔ No governed surface touched — .claude/**, skills/**, AGENTS.md, CLAUDE.md all absent from the diff, and Governed Surface Queue Guard is green. Remedy 2 was correctly found unnecessary rather than skipped.
  • verdict_head() is the single text site, and both verdict lines route through it — the locked one at :1718 and the declared-unlocked one at :1253. The two cannot drift apart, which was the stated reason for extracting it.
  • The ledger records $VERDICT_WORD, not a hardcoded command-exit, so an uncertified run is findable afterwards.
  • The scanner fails toward batch-last-exit by construction — backtick substitution and unbalanced quotes return 1 with a reason rather than being analysed. Over-labelling costs a warning; under-labelling would be this card's own defect. That asymmetry is the right one and it is documented as a choice.
  • The self-test pins both directions, which is what makes it a pin: a "fix" that labelled every run a batch would satisfy the mid-batch case alone while destroying the line's meaning. The &&-chain-that-fails case (command-exit 3) is the one that proves certification is not just "looks simple".
  • The fixture repair at :2373 — changing the filter-preflight live case from ; to && — is correct and not a workaround: that case asserts the certified word, so a ; would have made the fixture itself a batch.

Recorded, not held against it

The rejection of remedy 1 is the sharpest part of the header: this wrapper is handed one opaque string, so "aggregate the per-command exits" means writing a partial shell parser, and a partial parser that mis-read a ; inside a case arm would be an instrument answering confidently and wrongly — the very defect, shipped by its own fix. The point that "worst" is undefined over a set containing 99 = NOT MEASURED is independently right.

⚠️ The declared limit is the honest part and should stay quoted where it is: command-exit does not mean no failure can hide anywhere. An element that is itself a shell (bash -c 'a; b'), a subshell, or a pipeline inside an element still reports only its own last status. A future reader who over-reads the word will do so against an explicit refusal to claim it.

Landing conditions

  1. Lint & Repo Gates reports completed + success, read by name — currently in_progress.
  2. Nothing else. One file, no governed surface, no changeset owed (scripts/** ships in no package's files).

The ledger-identifiability half of the report is a PM matter, not work on this PR, and is being filed as its own finding rather than widened into this diff.


Generated by Claude Code

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/mskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants

@yinlianghui@claude