Fix flaky pty test: openpty(3) is not thread-safe on macOS

[jppittman]

What

io::pty_tests::pty_child_gets_default_sigpipe failed intermittently under cargo test --workspace while passing reliably when run alone. Root cause: openpty(3) is not thread-safe on macOS. It is not a timeout and not a SIGPIPE bug.

The tell is where it panics — pty_tests.rs:230 is the spawn_with_config(...).expect(...) line, not the read-with-timeout on 231. The 5-second READ_TIMEOUT is never reached and the SIGPIPE behaviour the test asserts is never exercised: openpty fails before a child is ever spawned. The test was a canary, not the culprit.

Evidence

Standalone C harness, openpty in a loop across N threads on a 12-core machine:

Mode	Threads	Runs	Total calls	Failures
unserialized	8	3	9,600	0
unserialized	48	3	57,600	107
serialized	48	3	57,600	0

It only reproduces once threads oversubscribe the cores — exactly what cargo test --workspace does with many test binaries running concurrently, and why it never showed up under -p core-term.

The failing calls return errno -6, a negative and therefore invalid errno. nix has no arm for it and falls through to _ => UnknownErrno, which is the signature observed in the wild. That detail also rules out the alternatives:

• Not pty exhaustion — kern.tty.ptmx_max is 511, exhausting it fails cleanly with a validENXIO, and only 5 were in use.
• Not fd exhaustion — ulimit -n is 1048576.
• Not the fork/malloc deadlock — that path is gone; spawn uses posix_spawn.
• Not a posix_spawn signal-attribute race — the failure precedes posix_spawn entirely.

Which internal call races is unconfirmed. ptsname(3) is the obvious suspect (man ptsname: not guaranteed reentrant or thread safe), but a corrupted replica path would make open() fail with a valid errno, not -6. The fix is justified by the measurement, not the mechanism — the doc says so explicitly rather than asserting a tidy story.

The fix — no production lock

Production opens exactly one PTY per run: main spawns it on the main thread before the troupe's threads exist, then hands it to PtyTroupe::new, which delivers it to the actors as a Bind message. Every other call site is inside #[cfg(test)], and there is no multi-window/multi-PTY design. So there is no live production race, and spawn_with_config now documents that single-threaded contract instead of carrying a lock for a race that cannot occur.

The contract is upheld in the test binary by a #[cfg(test)] OPENPTY_TEST_LOCK. It is gated inside spawn_with_config rather than sprinkled across the tests, because spawn sites live in four different test modules (io::pty_tests, io::event_monitor_actor::{mod, writer}, terminal_app) and a lock a new test can forget to take would let the flake back in silently. No integration test spawns a PTY, which is what makes cfg(test) a sufficient boundary. Poisoning is .expect()-ed so it fails loudly, per the project's no-silent-failures rule.

Deliberately not done: widening the timeout or retrying the spawn. Either would mask a genuine libc thread-safety bug behind a timing knob.

Reviewer notes

• In a non-test build the guard compiles away entirely — production behaviour is byte-for-byte unchanged.
• Considered and rejected, both recorded in the bug doc: a production Mutex (machinery for an impossible race, and Mutex<()> guards no data), and giving PTY spawning its own actor (the tidiest answer, but the natural home is the engine and pixelflow-runtime is deliberately generic while a PTY is terminal-specific).
• If a second PTY is ever needed for tabs or splits, the fix is to give spawning a single owner — not to make spawn_with_config reentrant. Noted in the rustdoc.
• nix::unistd::ttyname in the same spawn path is not a second instance of this bug; it already uses the reentrant ttyname_r.

Verification

• 7 full cargo test --workspace runs, zero failures and zero panics across all 74 test-result lines each.
• Targeted cargo test -p core-term --lib io::pty_tests passes.
• cargo build --workspace clean, no unused-import warnings.
• Pre-existing and unrelated: 3 clippy errors in pixelflow-runtime/src/platform/macos/cocoa.rs (raw-pointer lint), a file this PR does not touch.

🤖 Generated with Claude Code

[github-actions]

🤖 Hi @jppittman, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.