Skip to content

feat: retry transient Codex capacity failures - #33

Merged
RobertTLange merged 4 commits into
mainfrom
fix/codex-capacity-retries
Sep 10, 2026
Merged

RobertTLange merged 4 commits into
mainfrom
fix/codex-capacity-retries

Conversation

@RobertTLange

@RobertTLange RobertTLange commented Sep 10, 2026

Copy link
Copy Markdown
Owner

Problem and behavior

Codex can exit with Selected model is at capacity. Please try a different model. after starting a session. Headless previously returned that failure immediately, causing callers to exhaust task-level retries during the same capacity outage.

Automatically retry this exact native terminal failure up to three times, waiting 30/60/120 seconds with ±20% jitter. Preserve model, billing route, native session, and the original timeout; do not replay partial work without a session. HEADLESS_CAPACITY_RETRIES=0 disables retries, and values 1–3 set the limit.

Implementation

  • Keep capacity retries independent of the existing single billing fallback, with at most five executions and per-attempt usage provenance.
  • Emit structured retry events and stderr diagnostics; retain session storage across local, Docker, and Modal attempts.
  • Stop on timeouts, cancellation during backoff, and native/parent termination signals, including embedded callers whose child handles SIGTERM, with or without an explicit timeout.
  • Document defaults, recovery, configuration, and execution-mode coverage. Interactive/tmux launches remain native.

Validation

  • umask 0022; npm run check — build passed; 760 tests passed, 3 platform skips.
  • 50 focused core tests; affected core modules have 100% line and 97.24% branch coverage.
  • 39 backend tests and 7 existing signal/lifecycle tests passed. Fake agents cover resume, exhaustion, billing fallback, cancellation, Docker persistence, and Modal sandbox reuse; no live model calls.
  • Independent review plus local security, reliability, and contract checks; git diff --check passed.

The initial full-suite run under this host's umask 0002 failed nine existing Docker-session permission tests; rerunning with 0022 passed. No package release or Scientist image update is included.

@RobertTLange

Copy link
Copy Markdown
Owner Author

Fixed the cancellation review finding in 1b74091 (fix: preserve Codex cancellation without a timeout).

src/cli.ts:1921 now enables the existing parent signal handler for Codex independently of timeout configuration. An embedded caller forwarding SIGTERM to a child that catches it and exits 1 no longer triggers a capacity retry. Existing signal forwarding, external listeners, and cleanup remain intact.

Regression coverage in tests/capacity-cli.test.ts:160 exercises both timeout modes, checks one execution and no retry, verifies one delivery to the inherited listener, and confirms listener cleanup. The no-timeout case failed before the fix and passes afterward.

Validation:

  • node --import tsx --test tests/capacity-cli.test.ts tests/capacity-retry.test.ts: 34 passed.
  • umask 0022; npm run check: build passed; 760 tests passed, 3 platform skips.
  • git diff --check: passed; parallel review found no blockers.
  • CI run 34492191044: all six jobs passed on this commit.

PR description updated. No findings deferred. Validation used fake agents; no live model calls.

@RobertTLange
RobertTLange merged commit 7d4594a into main Sep 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant