Flaky daemon smoke test: findFreePort→bind race causes intermittent test-deno /health Connection refused

[taras]

Summary

The embedded smoke test packages/testing/tests/smoke.test.ts (golden EMBEDDED_TESTS) intermittently fails on the test-deno CI job (ubuntu-latest) at the "A daemon serves requests until its scope closes" chapter, defined in smoke-test/Guide/Daemons.md.

Failure signature:

TestFailureError: 1 of 47 tests failed
 A daemon serves requests until its scope closes: error sending request for url
 (http://127.0.0.1:<port>/health): client error (Connect):
 tcp connect error: Connection refused (os error 111)

Evidence that it is a pre-existing flake, not code-caused

The daemon chapter is byte-identical across three consecutive PR #136 commits, yet CI results differ:

commit	test-deno
d57f9eb	pass
47c467b	fail (this signature)
e7bce64	pass

Same test, same daemon chapter, different outcomes → flaky, unrelated to the PR's code. Locally deno task test passes 115 tests / 922 steps, and the daemon command runs fine (Node is present on the runner via the pre-installed ubuntu-latest toolchain; test-deno uses only setup-deno).

The chapter

smoke-test/Guide/Daemons.md does:

1. js eval — const daemonPort = yield* findFreePort()
2. bash daemon exec — node -e "require('http').createServer(...).listen({daemonPort},'127.0.0.1')"
3. js eval — when(() => fetch(daemonUrl + "/health").expect(), { timeout: 5000, interval: 50 })
4. curl + <AssertEquals … expected={"\ndaemon-ok"} />

Likely root cause

The classic findFreePort() → daemon-bind race: findFreePort probes a port, closes its socket, and returns the number; the node daemon then binds it. On a contended runner another process can grab the port in between → EADDRINUSE → the daemon exits → /health gets Connection refused for the full 5 s window → the chapter fails.

The daemon runs via daemonFactory (packages/core/src/modifiers/daemon.ts) using @effectionx/process daemon(cmd, { shell: true }), which backgrounds the process, so the child's stderr and exit status are not surfaced — hence the true cause is currently invisible in CI.

Suggested approach (do not just bump the timeout)

1. First capture the daemon child's exit status + stderr on the runner (confirm EADDRINUSE vs. a spawn/other failure) — e.g. redirect the node -e stderr to a file and surface it on when() failure, or add a one-off diagnostic step.
2. Fix the root cause (bind-with-retry on EADDRINUSE, or bind to port 0 and report the actual port instead of findFreePort + fixed-port), in a separate minimal PR against main.
3. Note: changing this chapter requires updating the EMBEDDED_TESTS golden list in packages/testing/tests/smoke.test.ts.

Surfaced while landing PR #136 (agent/acp-stack-07-acp-client); that PR is otherwise green and its failure was solely this flake.

[taras]

New recurrence observed while validating #279 / PR #284 (2026-08-02): after findFreePort() released its probe socket and before the daemon consumer bound, an unrelated local development server acquired the selected port. The daemon smoke then received that foreign application HTML. A rerun passed.

This confirms the existing TOCTOU diagnosis with a second externally observable outcome: the race can produce a valid response from the wrong process, not only connection refusal. Planner ruling: keep the fix out of #279 and address it on this issue so the concurrent-verification work does not widen scope.

[taras]

The same race, in the direction this issue doesn't describe

This issue records the race as connection refused — a port that should be
serving isn't. The concurrent battery on #313 tripped it the other way: a port
that should be dead answered.

packages/core/tests/provider-integration.test.ts — S5: provider crash during
when — daemon exits before ready. Its provider declares exit 0 as its daemon
block, so the port findFreePort() returned is never bound, and the readiness
poll is supposed to exhaust its 500 ms timeout:

yield* when(function*() {
 yield* fetch(baseUrl + '/health').expect();
}, { timeout: 500, interval: 50 });

Instead:

Expected pattern: /error|Error|ECONNREFUSED|timeout|Timeout/i
Received string: "·····
should-not-appear·
"

That received string is the healthy render — children expanded, no error —
so when() got a 2xx from 127.0.0.1:<port>/health while nothing in that
document was listening.

Why the concurrent battery makes it reachable

findFreePort() binds port 0, takes what the OS gives, and closes; nothing
reserves the port between the probe and the bind. Meanwhile S1–S4 in the same
file start

node -e "require('http').createServer((q,s)=>{s.writeHead(200);s.end('ok')}).listen({port},'127.0.0.1')"

which answers 200 on any path, /health included.

deno task verify (#313) runs deno task test, pnpm test:node, and
bun run test:bun simultaneously on one machine, so three copies of this suite
are probing and binding ports at the same time. A port one runtime's probe just
released can be bound by another runtime's provider daemon before the first
runtime's S5 fetches it — and S5 then sees a perfectly healthy provider.

Rate

At 092f465, seven completed runs of the concurrent battery: one failure.
Standalone pnpm test:node at the same head passed, as did the other six
batteries. It appears only under cross-runtime concurrency, which is consistent
with the mechanism above rather than with load alone — per-run timings were
stable (whole battery 5:10–6:05).

Not fixed by #313

Nothing in that PR touches findFreePort(); its plan puts this explicitly out
of scope. Recording it here because the issue currently describes only one
direction of the same missing reservation, and a reader hitting the "port that
should be dead answered" signature would not match it to this issue.