Skip to content

fix(scripts): gen-sdui-manifest cleanup reaps the session it started instead of leaking a flock-holding orphan - #9580

Merged
os-steve merged 2 commits into
mainfrom
claude/issue-9399-sdui-dump-orphan
Aug 18, 2026
Merged

fix(scripts): gen-sdui-manifest cleanup reaps the session it started instead of leaking a flock-holding orphan#9580
os-steve merged 2 commits into
mainfrom
claude/issue-9399-sdui-dump-orphan

Conversation

@os-steve

Copy link
Copy Markdown
Collaborator

Fixes#9399

The card's diagnosis was partly wrong, and the wrong half matters

The card inferred: "the wrapper re-execs, so $! is not the pid that matters." Measured here, that is false. /opt/node22/bin/pnpm is a node script with a #!/usr/bin/env node shebang, so bash execs it in the process it forked for the background job — $!is the node .../pnpm --filter @object-ui/console exec vite dev process, which is exactly the shape of the survivor in the incident's ps line. A fix aimed at "track the pid that actually binds the port" would have been aimed at a non-problem.

The second card observation, "ss -ltn showed nothing on 5180", is an artefact, not a measurement: ss and netstat are absent from these containers. The usual spelling hides it — ss -ltn | grep :5180 prints nothing and exits 0 whether or not anything is listening. So "it was no longer serving anything" is unproven. (The companion claim — that the vite child was gone — does come from ps and stands.) The wait loop is now commented to say why it uses curl, per docs/qa/platform-checklist/RUNNER.md.

What I could not determine, and why the fix does not depend on it

I could not reproduce the specific hang in which that particular SIGTERM failed to reap that particular run: it needs a real vite under a real console build tree, and .cache/objectui-* is absent from a fresh worktree. With stubs, plain SIGTERM to the pnpm wrapper reaps it cleanly every time, including under an early-signal race.

So the cleanup here is deliberately cause-agnostic: it escalates TERM to KILL, it verifies, and it prints what it could not kill. Picking one diagnosis and fixing only that is the failure mode where the leak looks fixed and recurs identically.

Three properties, each measured, each independently load-bearing

1. kill "$!" cannot reap the tree. Measured: after SIGTERM to the wrapper, a descendant survives reparented to init (PPID 1). No single-pid kill reaches it. Signalling the process group does.

2. The group could not be signalled safely before this change. A background job in a non-interactive shell does not get its own process group — it inherits the script's. Measured under the agent heavy-verify discipline, the backgrounded server's PGID was the PID of the wrapping flock itself. So the card's suggested kill -- -$PGID on the inherited group would have killed the caller's lock holder and the script. setsid is what makes a group kill bounded, and it is a prerequisite for the fix, not a flourish.

3. The fd inheritance is real, and it is what makes this a container outage.flock(1) holds its lock on an open fd; background children inherit open fds. Measured before the change: the backgrounded wrapper held the caller's lock fd and fuser -v listed it as a holder. Its own grandchild held zero — node closes the fd across its spawn — so the wrapper is precisely the process that converts "a leaked dev server" into "every later agent in this container queues out at exit 99". Measured after: fuser -v lists only flock/bash, the server is absent, and the lock is re-acquirable.

The triage comment asked for the fd half to be treated as a required half rather than an optional garnish. It is: it is the half that makes a missed kill survivable.

The change

  • Start the server via setsid, in its own session, with every inherited descriptor above stderr closed before exec.
  • Record the leader pid through a pidfile written by the child rather than $!. Here the two agreed (setsid exec'd in place instead of forking) — the pidfile is used because whether setsid execs or forks depends on job control, and the failure mode of guessing is a silently wrong pid.
  • Cleanup signals the session, escalates TERM to KILL, polls for confirmation, and on failure prints the surviving processes and says they may hold caller descriptors. Idempotent.
  • Trap INT/TERM/HUP as well as EXIT, re-raising after cleanup: bash runs no EXIT trap when the script is signalled, which is how an agent container reclaims a run.
  • The trap is armed before the spawn.

packages/spec/scripts/check-generated-ledger.test.ts asserts this script still contains check:react-declaration-parity; it does.

Pinning it — including one green line that was a lie

The new test sources the script (it returns right after defining its helpers, so no generation runs) and drives the real functions against a stub that reproduces the reparented-orphan shape.

Ablations, run against the final commit, each turning a different assertion red:

ablationresult
force the no-setsid pathred (session assertion)
delete the inherited-fd close loopred — LOCKFDS=1, expected 0
group kill to single-pid killred — 15876: expected '15876' to be '', and the script's own could not stop diagnostic fired

The middle row is the point of running ablations at all. On the first version of this test that ablation stayed green. The fd count ran after the stub leader had already exited, so it only ever inspected the reparented helper — which never inherits the descriptor anyway. It was a phantom check that would have shipped a permanently-unfailable assertion. Moving the count to while the leader is alive is what makes it real, and the test now asserts the leader was alive at the moment of counting so the ordering cannot silently regress. The orphan's prior existence is asserted too, so the survivor check can never pass vacuously.

Verification

All at 68bdc2ee1, the final commit; worktree clean.

  • pnpm --filter @objectstack/spec test410 files / 10938 tests passed
  • pnpm --filter @objectstack/spec typecheck — green, including check:test-typecheck (the new file compiles; debt held at 55 files / 263 errors, unchanged)
  • gates derived from the actual changed paths via node scripts/pm/dispatch-gates.mjs, which named a family the dispatch did not, because adding a test file moves gates the script path alone does not: check:nul-bytes, check:engine-double-contract, check:where-matcher, check:query-options-erasure, check:merge-driver, check:type-source-resolution, check:affected-docs, and spec check:empty-state / check:liveness / check:strictness-ledger / check:variant-docsall green
  • Not measured locally, deliberately: check:dev-prereqs, check:type-check-coverage, check:type-check-debt. All three require the whole workspace built; on this worktree check:dev-prereqs exits 1 reporting 67/67 packages missing dist/, which is the unbuilt-tree precondition and not a finding about this diff. Reported as not measured rather than green — CI runs them.

Scope

The dispatch scoped the file surface to scripts/gen-sdui-manifest.sh. The fix is entirely in that file. The one other path is a new test file, packages/spec/scripts/gen-sdui-manifest-cleanup.test.ts, which can collide with nothing because it did not previously exist; it is the regression pin the triage comment asked for. Flagging it rather than burying it.

No changeset: this changes repo tooling and a test, and releases nothing — the skip-changeset case named in lint.yml.

Out of scope: #9578, filed separately and remaining open — the script hardcodes port 5180 and one log path, so two concurrent runs in one container may dump each other's server.


Generated by Claude Code

The EXIT trap armed `kill "$DUMP_DEV_PID"`, ran, and left the dev server
alive 20 minutes later holding the container's shared heavy-verify flock, so
every later agent's build queued out at exit 99 with no signal.
Three measured properties of the old form were each independently wrong:
- `kill "$!"` does not reap the tree. Descendants reparent to init and no
single-pid kill reaches them; signalling the process GROUP does.
- The group could not be signalled safely: a background job in a
non-interactive shell inherits the SCRIPT's process group, which under the
heavy-verify discipline is led by the wrapping `flock` itself. `setsid`
makes the group kill bounded.
- `flock(1)` holds its lock on an open fd and background children inherit
open fds, so the orphan held the caller's lock. Closing inherited
descriptors makes a missed kill untidy rather than a container-wide stall.
Cleanup now escalates TERM to KILL, verifies, and reports what it could not
kill. Traps INT/TERM/HUP too, since bash runs no EXIT trap when signalled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
The lock-fd assertion was a phantom check: it ran after the stub leader had
exited, so it only ever inspected the reparented helper, which never inherits
the descriptor. Measured — with the fd hygiene deleted the test still passed.
Counting while the leader lives turns that ablation red (LOCKFDS=1).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
@claudeclaudeBot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

@os-steve
os-steve marked this pull request as ready for review August 18, 2026 12:44
@os-steve
os-steve added this pull request to the merge queueAug 18, 2026
Merged via the queue into main with commit b057e53Aug 18, 2026
29 checks passed
@os-steve
os-steve deleted the claude/issue-9399-sdui-dump-orphan branch August 18, 2026 13:02
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/mskip-changesetPR has no user-facing published change; bypasses the changeset gateteststooling

Projects

None yet

2 participants

@os-steve@claude