finding: `ObjectKernel`'s two `Promise.race` timeout guards leave a never-settled promise per boot and per shutdown — 4 async leaks on every showcase run

[os-zhuang]

Found while implementing #10293 / #10374, by running the showcase suite under vitest --detectAsyncLeaks (which #10374's triage grading proposed wiring as a gate). Recording only — ⛔ not fixed here, out of that card's scope.

What the detector reports

pnpm --filter @objectstack/example-showcase exec vitest run --detectAsyncLeaks on main @ b34ef8de8:

⎯⎯⎯⎯⎯⎯⎯ Async Leaks 4 ⎯⎯⎯⎯⎯⎯⎯⎯

PROMISE leaking in test/approval-resume-relation-expand.test.ts
 ❯ ObjectKernel.raceStartupTimeout ../../packages/core/src/kernel.ts:636:32
 ❯ ObjectKernel.initPluginWithTimeout ../../packages/core/src/kernel.ts:596:24
 ❯ ObjectKernel.bootstrap ../../packages/core/src/kernel.ts:356:28

PROMISE leaking in test/approval-resume-relation-expand.test.ts
 ❯ ObjectKernel.raceStartupTimeout ../../packages/core/src/kernel.ts:643:34

PROMISE leaking in test/approval-resume-relation-expand.test.ts
 ❯ ObjectKernel.shutdown ../../packages/core/src/kernel.ts:461:36

PROMISE leaking in test/approval-resume-relation-expand.test.ts
 ❯ ObjectKernel.shutdown ../../packages/core/src/kernel.ts:469:27

 Test Files 24 passed (24)
 Tests 364 passed (364)
 Leaks 4 leaks

Exit code is 0 — --detectAsyncLeaks reports, it does not fail the run.

The shape, and the asymmetry between the two sites

Both build a reject-only timeout promise and race it. Neither ever settles the losing promise, so it is retained past the end of the file — which is what the detector names. The two sites then diverge, each doing exactly one half of the cleanup:

• raceStartupTimeout (kernel.ts:629-647) clears the timer in a finally but never unrefs it.
• shutdown (kernel.ts:459-469) unrefs its timer but never clears it, so when performShutdown() wins the race the timer stays armed and fires later against a kernel that is already 'stopped'. That later rejection is handled — Promise.race already attached a rejection handler to that participant — so it is not an unhandled-rejection risk; it is retained work and a wakeup after teardown.

What this finding does NOT claim

⛔ These are not the trigger for the #10293 / #10374 console-teardown flake, and must not be read as its root cause. That flake needs a late onUserConsoleLog RPC, i.e. async work that emits console output after the test file ends. These four promises emit nothing. They are named here because they are what --detectAsyncLeaks actually finds in this suite today — a fact that matters for #10374's proposal to wire that flag as a gate, since its current output on the showcase is four findings that cannot cause the defect it was proposed to catch.

Measured cost of the flag, for whoever takes that decision: showcase suite 21s → 40s wall (~1.9x), same 24 files / 364 tests.

Suggested shape if it is taken

Give both sites the same treatment — settle or drop the loser, and both unref and clear the timer — and keep it in one helper rather than two hand-rolled races, so the two halves cannot drift apart again.

---

Generated by Claude Code

[claude]

Claim: /pm-dispatch domain:devx execution seat.
Session: session_01DdCnBGcHeufjrq7drTD3wt
Branch: claude/issue-10604-kernel-race-timeout-leaks
Worktree: objectstack-issue-10604
Domain: domain:devx
File surface: packages/core/src/kernel.ts (raceStartupTimeout, initPluginWithTimeout, shutdown) + a pin for the fixed behaviour

⚠️ This one touches published package source, unlike most of tonight'''s round — the changeset judgement is genuinely open and the dev is asked to make it explicitly rather than default to skip-changeset.

⛔ Governed surface out of scope (docs/adr/**, .claude/**, skills/**, AGENTS.md, CLAUDE.md, content/docs/releases/**) and the #9465 epic fence (.changeset/ tooling, cut-rc.yml, release.yml, root package.json).
⚠️ This repo has no formatter of record (no format/prettier script, prettier not in devDeps, no .prettierrc/.editorconfig) — ⛔ do not run prettier --write (#10622).

---

Generated by Claude Code

[claude]

{
 "issue": 10604,
 "status": "done",
 "branch": "claude/issue-10604-kernel-race-timeout-leaks",
 "pr": "https://github.com/objectstack-ai/objectstack/pull/10661",
 "premise_still_valid": true,
 "summary": "Both ObjectKernel timeout guards now go through one internal TimeoutGuard (packages/core/src/timeout-guard.ts) whose reclaim() does BOTH halves: clearTimeout AND settling the promise Promise.race still holds a reaction on. Settling the loser is what removes the leak -- clearing the timer alone removes none of the four frames, because two of them are the race's own reaction promises on the never-settling participant. DEVIATION FROM THE BRIEF, deliberate: the brief and the card both say 'clear AND unref at both sites'. I did not add unref; I REMOVED the one at the shutdown site. #4813 already landed the opposite finding in this same file -- an unref'd guard stops pinning the loop but stops being a guard too, so a hung teardown in an otherwise-idle process exits silently at status 0 and never reaches 'Shutdown timed out -- forcing exit'/exit(1). Reclaiming on settle gets the loop hygiene unref was reaching for without giving up the guarantee, so the two goals were never in tension. Ablation C below is the pin that catches the brief's version. THREE MORE BRIEF/CARD CLAIMS THAT DID NOT HOLD, none blocking: (1) 'Re-derive the line numbers; kernel.ts may have moved since b34ef8de8' -- it had NOT moved. raceStartupTimeout is still 629-647 and shutdown still 457-469, and all four frames reproduced at exactly 636:32 / 643:34 / 461:36 / 469:27. (2) 'construct a plugin that exceeds the startup timeout and one whose shutdown hangs' -- both already exist as landed pins ('hanging-plugin' #4813, 'hanging-shutdown-plugin' #5274) and are green unchanged, so the timeouts-still-fire proof was already in the tree. I wrote a duplicate of the shutdown one before finding #5274's and deleted it in the second commit rather than ship two near-identical tests. (3) the DoD's per-package `pnpm typecheck` does not exist for @objectstack/core (scripts are build/test only); `pnpm --filter @objectstack/core typecheck` fails loudly with ERR_PNPM_RECURSIVE_RUN_NO_SCRIPT, not the silent zero-match exit-0 trap. Running `tsc --noEmit -p tsconfig.json` there surfaces 98 PRE-EXISTING errors, all in *.test.ts and all already ledgered (scripts/check-type-check-coverage.mjs records errors: 98 for @objectstack/core); that is debt-ledger territory, not a verdict on this change. I used an explicit './timeout-guard.js' specifier in the new test precisely so it adds ZERO debt -- the sibling test files omit the extension and each pays one TS2835; with the bare specifier my file made it 99 and the check:type-check-debt ratchet would have gone red. It is green at 98.",
 "tests": "All heavy steps via scripts/pm/os-verify-lock.sh; verdict lines quoted, never a bare $?. BEFORE (pristine tree, merge-base 47aff09388, `git status --porcelain` empty): `pnpm --filter @objectstack/example-showcase exec vitest run --detectAsyncLeaks` -> 'Async Leaks 4', frames raceStartupTimeout kernel.ts:636:32 + 643:34 and shutdown kernel.ts:461:36 + 469:27, 'Test Files 24 passed (24) / Tests 364 passed (364) / Leaks 4 leaks'. AFTER at final head 63ea716fb4: 0 occurrences of 'PROMISE leaking', 0 'Async Leaks' sections, 'Test Files 24 passed (24) / Tests 364 passed (364)' -- baseline unmoved. The showcase resolves @objectstack/core through the workspace link to dist/, so `pnpm --filter @objectstack/core build` runs between the two measurements. CORE SUITE at 63ea716fb4: 'Test Files 37 passed (37) / Tests 887 passed (887)'. TIMEOUTS STILL FIRE (verbose reporter, post-fix): '(check) #4813 > still fires the guard when the plugin loses the race 50ms'; '(check) #5274 > still logs the timeout and still forces exit(1) when shutdown genuinely times out 22ms'; '(check) still rejects with the caller's own error object when it is never reclaimed 11ms'; '(check) arms a REF'D timer, so an otherwise-idle process cannot exit before it fires 0ms'. ABLATIONS -- three, each mutation confirmed ON DISK BY ANCHOR COUNT before the run and each restore confirmed the same way; no editor exit code was used as evidence. Core's tests import ./kernel and ./timeout-guard.js as SOURCE (colocated *.test.ts, no dist in the resolution path), so no rebuild leg applies to these three -- the rebuild discipline does apply to the showcase leak runs above and was followed there. (A) settle half removed: `this.settleExpiry();` count 1->0, marker 1 -> 2 red, exactly the two settlement pins, 'Tests 2 failed | 53 passed (55)'. Restore: count back to 1, marker 0. (B) clear half removed: `clearTimeout(this.timer);` count 1->0, marker 1 -> 7 red including the new '#10604 > leaves no timer armed once shutdown has settled' AND the pre-existing #4813 pins (which is the evidence the shared helper really is on the startup path too), 'Tests 7 failed | 48 passed (55)'. Restore: count back to 1, marker 0. (C) unref() at arm time -- i.e. the brief's own suggestion: `this.timer.unref()` count 0->1 -> 1 red, and ONLY the anti-disarm pin; every leak pin stayed GREEN. That is the point: 'fix the leak by disarming the guard' passes every leak assertion. Restore: count back to 0, marker 0, then 'Test Files 2 passed (2) / Tests 55 passed (55)'. GATES: union derived with `node scripts/pm/dispatch-gates.mjs` with NO paths passed (it took the change set from merge-base itself and named exactly my 5 files). 10 path-matched + 5 convention-triggered + check:nul-bytes = 16 families, ALL exit 0 at head 63ea716fb4, after the turbo build the type-check-debt ratchet requires: nul-bytes, changeset-gate-self-tests, kernel-hook-pairs, objectui-changeset, slot-lookup, test-source-alias, type-source-resolution, check-adr-0087-registration, check-changeset-no-major, check-empty-changeset, docs-audit/check-affected-docs, query-options-erasure, type-check-coverage, type-check-debt, engine-double-contract, where-matcher. eslint on all four changed source files: exit 0, no output. No prettier --write (#10622). Control-byte self-scan over all five changed files: no matches.",
 "open_questions": [],
 "out_of_scope_findings": []
}

---

Generated by Claude Code

[claude]

{
 "issue": 10604,
 "status": "done",
 "branch": "claude/issue-10604-kernel-race-timeout-leaks",
 "pr": "https://github.com/objectstack-ai/objectstack/pull/10661",
 "premise_still_valid": true,
 "summary": "The posted diagnosis was correct and I confirmed it, then narrowed it one step further. The one red of 28, Test Core (4/6), was this PR's own new pin scoring itself against process.getActiveResourcesInfo(), a PROCESS-wide count that 37 co-tenant test files share in one CI worker; `expected 2 to be 4` is the count going DOWN, which no leak can cause. THE NARROWING THE BRIEF DID NOT HAVE: the CI log's frame is timeout-guard.test.ts:156:24, i.e. the THIRD leg specifically -- the only one that spends real time on the loop. Legs 1 and 2 settle on microtasks, where no timer phase can run, so the reading provably cannot move under them, and they passed on exact equality in the same failing run. REPAIR (1 file, 26 insertions / 4 deletions, packages/core/src/timeout-guard.test.ts): each leg re-anchors on its own sample and asserts toBeLessThanOrEqual. I took the brief's preferred shape and added per-leg re-anchoring so a comparison spans one outcome instead of the whole test. I considered and REJECTED the brief's alternative of a probe the test owns outright (vi.useFakeTimers + vi.getTimerCount): the sibling pin at timeout-guard.test.ts:34-47 already makes exactly that assertion, so converting this one would collapse it into a near-duplicate of a pin the prior seat had just finished de-duplicating, and would drop the ref'd-ness dimension that is this pin's only reason to use the real probe. Nothing in the runner or shard configuration was touched. FILE: /home/user/objectstack-10661-fix/packages/core/src/timeout-guard.test.ts, committed as e5582bf603 and pushed to the same branch; PR left draft, labels and auto-merge untouched.",
 "tests": "All heavy steps through scripts/pm/os-verify-lock.sh; verdicts quoted, and every exit code captured by redirect-then-capture, never after a pipe. GATE UNION AND FINAL HEAD ARE THE SAME TREE: e5582bf603 (git rev-parse --short HEAD taken from the gate run itself; no commit after it). ### REPRODUCTION -- could NOT reproduce naturally, said plainly. `pnpm --filter @objectstack/core test` = 'Test Files 37 passed (37) / Tests 887 passed (887)', and `vitest run --no-file-parallelism` (all 37 files in ONE worker, the CI condition) green three more times: 4 green runs total, zero flakes. So I built the condition synthetically rather than claim a fix I never saw fail: a scratch setupFiles arming ambient ref'd timers at 1ms spacing over 3s (what 37 co-tenant files look like to a process-wide probe), run via a scratch vitest config. Result: 'FAIL src/timeout-guard.test.ts > raceWithTimeout (#10604) > leaves no ref'd timer behind on any of the three outcomes / AssertionError: expected 2872 to be 2885 / timeout-guard.test.ts:156:24 / Tests 1 failed | 10 passed (11)' -- same test, same LINE, same downward shape as CI. Note what did NOT fail under ~2885 timers expiring at ~1/ms: the other 10 pins, including the synchronous anti-disarm ref'd pin AND legs 1-2 of this very test on exact equality. That is the empirical proof of the microtask-window claim. AFTER the repair the same command is green 5/5, and the suite is unmoved: 37 files / 887 tests, both default and --no-file-parallelism. Both scratch files were archived to the scratchpad and DELETED from the worktree before deriving gates (`git status --porcelain` empty). ### ABLATIONS -- all three re-run against the REPAIRED pin. Core's tests import ./timeout-guard.js and ./kernel as SOURCE (colocated *.test.ts, no dist in the resolution path, no vitest alias in packages/core/vitest.config.ts), so no rebuild leg applies to these three; the rebuild discipline was applied where it does bite (the turbo build below, and the dep closure `pnpm --filter '@objectstack/core^...' build` which the first run needed -- without it 12 files fail COLLECTION on `Cannot find package '@objectstack/spec/system'`, which reads nothing like a missing build). The fix was COMMITTED before ablating, so every restore was `git checkout -- src/timeout-guard.ts` from a real recovery point. Each mutation and each restore confirmed ON DISK BY ANCHOR COUNT by a driver that refuses unless the anchor matches exactly once and then re-reads the file to count the removed/injected text; no editor exit code was used as evidence. (A) settle half removed -- `this.settleExpiry();` 1->0 confirmed on disk -> 2 red, exactly the two settlement pins, 'Tests 2 failed | 52 passed (54)'. (B) clear half removed -- `clearTimeout(this.timer);` 1->0 confirmed -> 7 red, 'Tests 7 failed | 47 passed (54)', and THE REPAIRED PIN IS ONE OF THE SEVEN, alongside the pre-existing #4813 pins. (B') the same ablation under the synthetic ambient noise -> STILL RED, 'Tests 3 failed | 8 passed (11)', repaired pin included: a real leak is caught with ~2885 foreign timers expiring at ~1/ms, which is the direct answer to 'prove the pin still catches a real leak'. (C) `this.timer.unref();` at arm time -- 0->1 confirmed -> 1 red, and ONLY the anti-disarm pin; every leak pin including the repaired one stayed GREEN. The asymmetry is undisturbed. Restore legs: counts back to settle=1 clear=1 unref=0 each time, and the restored tree is 'Test Files 2 passed (2) / Tests 54 passed (54)'. ### GATES. Union re-derived with `node scripts/pm/dispatch-gates.mjs` and NO paths passed; it printed 'change set derived from git -- 5 path(s) vs merge base 47aff0938 (committed 5, working tree 0, untracked 0; three-dot semantics)' and named the same 10 path-matched + 5 convention-triggered families as the prior seat -- my commit adds no new path, so the union is unchanged; +check:nul-bytes = 16. After `pnpm exec turbo run build --concurrency=2 --filter='./packages/*' --filter='./packages/*/*'` ('Tasks: 70 successful, 70 total'), ALL 16 EXIT 0 at e5582bf603: nul-bytes, changeset-gate-self-tests, kernel-hook-pairs, objectui-changeset, slot-lookup, test-source-alias, type-source-resolution, adr-0087-registration, changeset-no-major, empty-changeset, affected-docs, query-options-erasure, type-check-coverage, type-check-debt, engine-double-contract, where-matcher. Their own verdict lines, not my $?: 'check-type-check-coverage --re-measure: OK -- 33 ledger entr(ies) re-measured in 269.1s, 1913 raw tsc error(s) total, none above its recorded number' (the ratchet the brief flagged: @objectstack/core's ledger entry is errors: 98 and nothing is above its recorded number; I did not touch the './timeout-guard.js' specifier); 'check-nul-bytes: OK (scanned 6214 text file(s) ... no raw ASCII control bytes)'; 'check-test-source-alias OK -- 72 packages with tests scanned'; 'check-engine-double-contract: OK -- 371 pinned'. eslint on the changed file: exit 0, no output; no prettier --write (#10622). Control-byte self-scan over the changed file: no matches. ### CI at report time: 31 checks on e5582bf603 -- 13 success, 2 skipped, 16 in_progress, ZERO red. All six Test Core shards still running, so the gate that matters is in_progress and I am not waiting on it per the report-at-draft-PR contract.",
 "open_questions": [],
 "out_of_scope_findings": [
 "filed as #10685: three ref'd-timer pins (kernel.test.ts #4813 x2, hot-reload.test.ts:80) score the same process-global count across an `await`; green today only because those awaits resolve on microtasks. Labelled `finding`, no `pm:queue`, unassigned -- not fixed, because the brief scoped this seat to the CI failure and nothing else."
 ]
}

---

Generated by Claude Code