Skip to content

Make the Codex probe's timeout actually bound doctor - #629

Merged
jeremy merged 4 commits into
mainfrom
fix/codex-probe-waitdelay
Aug 6, 2026
Merged

Make the Codex probe's timeout actually bound doctor#629
jeremy merged 4 commits into
mainfrom
fix/codex-probe-waitdelay

Conversation

@jeremy

@jeremyjeremy commented Aug 6, 2026

Copy link
Copy Markdown
Member

Found while cutting v0.9.0: make release VERSION=0.9.0 DRY_RUN=1 failed race-test with

panic: test timed out after 10m0s
running tests:
TestDoctorCommandWithNoAuth (9m47s)

Not a flake, and not a test bug. basecamp doctor really does hang.

What happens

queryCodexPlugin has had a 5s context timeout since it was written. The timeout was doing nothing.

Canceling the context kills the child. It does not close output pipes a grandchild inherited, and Wait blocks on those copies until they close. The stack shows exactly that — one goroutine parked in io.Copy on the pipe, another in exec.(*Cmd).awaitGoroutines, both nine minutes in:

os/exec.(*Cmd).awaitGoroutines
os/exec.(*Cmd).Wait
os/exec.(*Cmd).Output
harness.init.func1 (internal/harness/codex.go:30)
harness.queryCodexPlugin
harness.CheckCodexPluginDiagnosticsContext
commands.runDoctorChecks

On the machine that caught it, codex is a wrapper script rather than a binary. The processes were still there afterwards:

2236158 13:56 /bin/bash ~/.local/bin/codex plugin list --available --json
2236159 13:56 npm exec which @openai/codex

The kill reached the wrapper. npm exec kept the inherited stdout pipe open, so Wait waited on it. The deadline expired on schedule and the call returned whenever the grandchild felt like it.

This is not specific to that launcher — an npm-exec shim, a mise shim, anything that shells out has the shape. It never reproduces in CI, where codex is absent and FindCodexBinary returns early.

The fix

cmd.WaitDelay bounds Wait itself, not just the process — the one piece that was missing. A codex that will not answer inside the deadline is now a failed check, which is what a diagnostic probe should report.

Both durations are named constants now; the timeout and the grace period after the kill belong next to each other.

internal/harness/codex.go:30 is the only exec on the doctor's probe path — the Claude probe reads files — so this is the whole exposure there. Nothing else in the repo sets WaitDelay; the other 24 call sites are interactive setup and upgrade paths, and widening to them is a separate change with its own tests, not something to fold into a release fix.

Verification

Every existing test in this file stubs runCodexCommand, so none of them ever touched the real one. The new test does, against a script that backgrounds a longer-lived child:

Result
without WaitDelayFAIL at 30.00s — "did not return: WaitDelay is not bounding Wait"
with WaitDelayok, 1.3s

Verified by reverting the one line and re-running, not by inspection.

And the test that actually hung, re-run on the same machine with the same wrapper still installed:

$ go test -tags dev -race -run 'TestDoctorCommandWithNoAuth|TestRunCodexCommandOutlivingGrandchild' ./internal/commands/ ./internal/harness/
ok .../internal/commands	2.015s
ok .../internal/harness	2.012s

9m47s and a timeout, to 2s. bin/ci green (exit 0).


Summary by cubic

Fixes a hang in basecamp doctor by making the Codex probe truly time out using cmd.WaitDelay, and strengthens the linter lockstep check to prevent stale or missed pins.

  • Bug Fixes
    • Doctor Codex probe now respects its deadline: set cmd.WaitDelay (1s) and use codexQueryTimeout (5s) so Wait can’t block on a grandchild’s stdout pipe.
    • Added a test that simulates a wrapper spawning a long-lived child; it fails without WaitDelay and passes with it. The test now records the grandchild PID and reaps it in t.Cleanup, and uses sh found via exec.LookPath.
    • Updated scripts/check-lint-lockstep.sh to scan both .yml and .yaml workflows, fail on empty scans, and enforce a minimum golangci-lint version using version-aware comparison.

Written for commit 6995cd6. Summary will update on new commits.

Review in cubic

Two holes in what shipped, both the same shape as the drift the check
was written for.
It globbed *.yml only. Every workflow here is .yml today so nothing was
skipped, but GitHub honours .yaml just as well, and a .yaml workflow
running a linter this check never opened is exactly the invisible drift
it exists to catch. Scan both, and fail on an empty scan rather than
passing a check that opened no files.
It enforced agreement, not currency. Setting all three workflows to
v2.9.0 passes cleanly and reproduces the release failure that motivated
the check in the first place. So: a floor, next to WORKFLOW_DIR, moved
in the same commit as the pins it constrains.
Compare with sort -V. As strings v2.9.0 sorts above v2.11.1, because
9 > 1, so a lexical test would wave through the precise version that
broke the release tag — and [ -gt ] parses neither. aur-publish.yml's
pkgrel guard carries the same note for the same reason.
`basecamp doctor` hung for ten minutes on a machine where `codex` is a
wrapper script. queryCodexPlugin has had a 5s context timeout since it
was written, and the timeout was doing nothing.
Cancelling the context kills the child. It does not close output pipes
a *grandchild* inherited, and Wait blocks on those copies until they
close — so `codex` shipped as a launcher that shells out (an npm exec
wrapper, a mise shim) leaves Wait parked on a pipe held open by a
process the kill never reached. The deadline expires on schedule and
the call returns whenever the grandchild feels like it.
cmd.WaitDelay is the part that was missing: it bounds Wait itself, not
just the process. A codex that will not answer inside the deadline is
now a failed check, which is what a diagnostic probe should report.
The existing tests all stub runCodexCommand, so none of them touched
the real one. The new test does, against a script that backgrounds a
longer-lived child — it fails at 30s without WaitDelay and passes in
about a second with it.
Both durations are named constants now; the timeout and the grace
period after the kill belong next to each other.
CopilotAI balanced review requested due to automatic review settings August 6, 2026 01:01
@github-actionsgithub-actionsBot added the tests Tests (unit and e2e) label Aug 6, 2026
CopilotAI reviewed Aug 6, 2026

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:2a8b47c316

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threadscripts/check-lint-lockstep.sh
Comment threadinternal/harness/codex_test.go Outdated
The test needs a grandchild that outlives the deadline by a wide
margin, or it passes on the sleep ending rather than on WaitDelay
working — so the 120s is load-bearing and cannot just be shortened.
That makes reaping the test's job. WaitDelay closes the inherited
pipe; it does not kill the process, which is reparented to init and
sat there for two minutes afterwards, one orphan per `bin/ci`.
The shell now records the background PID and t.Cleanup kills it,
including on the failure path. Also runs the `sh` that LookPath
resolved rather than assuming /bin/sh, since the skip already
depends on that lookup.
CopilotAI review requested due to automatic review settings August 6, 2026 01:10
CopilotAI reviewed Aug 6, 2026

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:6995cd6b29

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threadinternal/harness/codex.go
@jeremy
jeremy merged commit 37880ce into mainAug 6, 2026
24 of 25 checks passed
@jeremy
jeremy deleted the fix/codex-probe-waitdelay branch August 6, 2026 01:20
jeremy added a commit that referenced this pull request Aug 22, 2026
…or-roundtrip
* origin/main: (96 commits)
ci: bump the github-actions group with 6 updates (#639)
Reject three more doomed invocations before draining stdin (#645)
Stdin `-` support everywhere sensible; usage error for stray `-` elsewhere (#641)
Add hey-cli Windows signing secrets to the release env manifest (#642)
deps: bump the go-dependencies group with 5 updates (#638)
Update nix flake and plugin version for v0.9.1
ci: bump the github-actions group with 4 updates (#633)
Add basecamp files replace: publish a new version of an uploaded file (#634)
Add basecamp files versions — HELD, blocked on the SDK (#622)
Update nix flake and plugin version for v0.9.0
Make the Codex probe's timeout actually bound doctor (#629)
Make the lockstep check catch stale agreement and .yaml workflows (#628)
Keep refreshing opencode's other spelling (#627)
Lint the release the same way we lint everything else (#625)
Install the skill where opencode actually looks (#624)
Take the communiques out of the source tree (#623)
Correct the API coverage claim: 183/184, not 100% (#621)
Stop echoing back step fields the caller never changed (#620)
Drive the circuit breaker's clock from tests, not sleep() (#619)
Tell agents the truth about card column moves (#618)
...
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testsTests (unit and e2e)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@jeremy