Skip to content

fix(sandbox): bound the Daytona calls that requests wait on - #464

Merged
thesujai merged 4 commits into
truefoundry:mainfrom
ethanstoner:fix/daytona-request-timeouts
Aug 27, 2026
Merged

fix(sandbox): bound the Daytona calls that requests wait on#464
thesujai merged 4 commits into
truefoundry:mainfrom
ethanstoner:fix/daytona-request-timeouts

Conversation

@ethanstoner

@ethanstonerethanstoner commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Fixes#417.

What hangs, and where

checkSnapshotStatus reaches Daytona with no cap on the round-trip, from three request-handling paths:

  • GET /api/v1/capabilities — the UI's boot probe
  • GET /api/v1/settings/sandbox-providers — the settings page
  • turn creation, when a fresh sandbox is needed (turns.ts)

A provider that accepts the connection and then stalls holds each of those until undici's own multi-minute defaults, once per request, each keeping a server connection and an event-loop task alive. A Daytona brownout turns into piled-up concurrent requests, and nothing is logged while it happens.

Only PUT /settings/sandbox-providers bounded it, at withTimeout(provider.buildImage(), 3s).

What this does

One budget, one owner.BUILD_REQUEST_TIMEOUT_MS moves out of sandboxProviders.ts and becomes SANDBOX_BUILD_REQUEST_TIMEOUT_MS next to the refresh; the PUT handler imports it. Both paths issue the same calls, and a duplicated number is one edit from disagreeing — the one-canonical-owner rule in AGENTS.md.

The refresh returns the persisted status when the budget expires, rather than throwing. All three callers are asking what the status is, not asking to reach Daytona:

  • the capability probe already catches and fails closed, so a throw would report the sandbox unconfigured on a three-second blip
  • the settings read already falls back to the record
  • turn creation would take an exception it has no better answer for

A stale status serves each of them better than a hang or a 500 while the provider is unwell. A real error — rejected credentials, a failed build — still propagates, so this hides nothing except slowness. There is a test for that distinction.

Registration carries its own request timeout. This is the part worth a second look: withTimeout is a Promise.race, so it abandons the result while the request keeps running. A caller-side cap alone leaves the socket open for the full undici default, which is most of what the issue is about. AbortSignal.timeout on the fetch ends the request itself. Ten seconds there, deliberately looser than the caller's three: it is the backstop for a socket nobody is waiting on any more, not the budget a request is held to.

Verified

The two stall cases were run against the code with the withTimeout calls removed — both hang and fail on a four-second test timeout — and pass with them. The tests fail without the fix:

without the fix with it
× stalled status read (4004 ms) √ (38 ms)
× stalled reactivation (4004 ms) √ (4 ms)
√ real failure still surfaces √
√ status within the budget √
  • tests/unit/apis/ + tests/unit/sandbox/153 passed, 4 of them new.
  • tsc --noEmit clean for packages/trueforge src, its tests/unit project, and packages/trueforge-core.
  • prettier --check and eslint clean on the files touched.
  • Six failures elsewhere in packages/trueforge (LocalSandboxProvider, Code Mode UDS) are a Windows host — unix sockets and symlink paths — and fail identically on a clean checkout.

Deliberately not addressed

The issue's closing note: checkSnapshotStatus mutates persisted status and can reactivate an idle snapshot from GET handlers, so plain reads drive external writes and retrying clients amplify call volume. That is real, but it is a question about where the refresh belongs rather than a timeout, and fixing it would move behaviour this change is trying to leave alone. Happy to open a separate issue or PR for it if you want it split that way.

.changeset/daytona-request-timeouts.md covers both published packages.


Note

Medium Risk
Changes how sandbox status refresh behaves under slow or stuck Daytona responses on several API paths; timeouts surface as errors rather than indefinite hangs, which can affect caller error handling (e.g. settings GET vs capabilities fail-closed).

Overview
checkSnapshotStatus no longer waits indefinitely on Daytona when it calls buildImage() (ready snapshots that may need reactivation) or getImageBuildStatus() (non-ready builds). Both round-trips are wrapped with withTimeout from trueforge-core, using a new STATUS_REFRESH_TIMEOUT_MS of 60 seconds.

That refresh runs on request paths such as capabilities boot, sandbox settings GET, and turn creation when status must be refreshed—so a stalled Daytona provider stops holding those handlers for undici’s long default instead of hanging until the client gives up.

A patch changeset notes the behavior for @truefoundry/trueforge and @truefoundry/trueforge-core.

Reviewed by Cursor Bugbot for commit 3474dee. Bugbot is set up for automated code reviews on this repo. Configure here.

The sandbox build-status refresh talks to Daytona and had no cap on the round-trip. It runs on
GET /api/v1/capabilities, on GET /api/v1/settings/sandbox-providers, and on turn creation when a
fresh sandbox is needed, so a provider that accepted the connection and then stalled held those
requests until undici's own multi-minute defaults — once per request, with each one keeping a
server connection and an event-loop task alive. A Daytona brownout became piled-up concurrent
requests, and nothing was logged while it happened.
Only the settings write bounded this, wrapping buildImage in withTimeout at three seconds. That
budget now lives beside the refresh as SANDBOX_BUILD_REQUEST_TIMEOUT_MS and both paths import it:
they issue the same calls, and a duplicated number is one edit away from disagreeing.
On expiry the refresh returns the last persisted status instead of throwing. All three callers are
asking what the current status is, not asking to reach Daytona: the capability probe already fails
closed on an exception and would report the sandbox unconfigured, the settings read already falls
back to the record, and turn creation would take an exception it has no better answer for. A stale
status serves each of them better than a hang or a 500 while the provider is unwell. A real error —
rejected credentials, a failed build — still propagates, so this does not hide anything except
slowness.
Snapshot registration also carries its own timeout now. withTimeout is a Promise.race: it abandons
the result but the request keeps running, so a caller-side cap alone leaves the socket open for the
full undici default. AbortSignal.timeout ends the request itself.
Verified: the two stall cases hang and fail on a four-second test timeout with the withTimeout calls
removed, and pass with them, so the tests fail without the fix. tests/unit/apis and
tests/unit/sandbox: 153 passed. tsc clean for trueforge src, its tests/unit project, and
trueforge-core. Six failures elsewhere in the package are a Windows host — unix sockets and symlink
paths under LocalSandboxProvider — and fail identically on a clean checkout.
Not addressed here: the issue also notes that checkSnapshotStatus mutates persisted state and can
reactivate a snapshot from GET handlers, so reads drive external writes. That is a design question
about where the refresh belongs rather than a timeout, and changing it would move behaviour this
change is trying to leave alone.
@changeset-bot

changeset-botBot commented Aug 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: b91b87e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
NameType
@truefoundry/trueforgePatch
@truefoundry/trueforge-corePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@thesujaithesujai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We wanted timeout of the calls outside the transaction block to be 1 min

Comment threadpackages/trueforge-core/src/core/sandbox/provider/DaytonaProvider.ts Outdated
Comment threadpackages/trueforge/src/apis/sandboxProviders.ts Outdated
Comment threadpackages/trueforge/src/sandbox/providerUtils.ts Outdated
Comment threadpackages/trueforge/src/sandbox/providerUtils.ts Outdated
Comment threadpackages/trueforge/tests/unit/sandbox/providerUtils.test.ts Outdated
Comment threadpackages/trueforge-core/src/core/sandbox/provider/DaytonaProvider.ts Outdated
The refresh runs outside a transaction, so its budget is a minute rather than
the three seconds the settings write uses inside one. The write is untouched
and keeps its own constant.
On expiry it throws instead of returning the persisted status. Returning it
would report 'ready' for a snapshot that may have been deactivated, which is
the case the buildImage branch exists to catch. getCapabilities already treats
a failure as sandbox-disabled.
Registration's request timeout matches the longest caller budget, so the socket
outlives no one still waiting on it.
Comments trimmed and the test file removed, as asked.
@ethanstoner

Copy link
Copy Markdown
ContributorAuthor

Pushed. Summary of what changed:

  • 1 min outside the transaction. The refresh now uses a 60s budget of its own. The settings write inside the transaction is untouched and keeps its own 3s constant — I had renamed and shared it, which made the diff read as a change to the PUT. sandboxProviders.ts is byte-identical to main again.
  • Throws on expiry instead of returning the persisted status. You are right that returning ready there would assert the very thing the buildImage branch is checking for.
  • Comments trimmed, test file removed.
  • Registration timeout is 60s, matching the longest caller budget rather than the arbitrary 10s.

One thing I pushed back on rather than silently changing, with measurements in the thread: withTimeout frees the caller but does not stop the request, so the socket stays open until undici's default — that is the connection pile-up in #417 that a caller-side cap cannot reach. It is one line and I will drop it if you would rather keep this PR to the caller side.

Verified after the changes: tests/unit/apis 135 passed, tsc clean across trueforge src, its tests/unit project, and trueforge-core, prettier and eslint clean.

@thesujaithesujai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ethanstoner for your work here, really appreciate it!

@thesujai
thesujai merged commit c40129c into truefoundry:mainAug 27, 2026
8 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daytona snapshot registration has no request timeout — GET /capabilities, GET settings, and turn creation can hang ~5 min when Daytona stalls

2 participants

@ethanstoner@thesujai