Uh oh!
There was an error while loading. Please reload this page.
Agent harness: a bounded tool-calling loop with the WebContainer as control plane - #37
Merged
Conversation
Plans the replacement of Build's single-shot {reply, patches[]} agent with a
bounded tool-calling harness using the WebContainer as the control plane.
Research findings that shaped the design:
- project.files is the source of truth; the container FS is a lossy replica, so
every agent write must route through project.FileApplied or the editor,
publish, autosave, and next-turn context all go stale.
- The envVars/.env channel is dead code — no Gleam effect field, no server
request field, env-store.ts orphaned. Resolved by denying .env rather than
gating it.
- The 10/min rate limiter would 429 the user's second turn once one turn is
12 requests.
Plan gate returned NOT APPROVED with 12 blockers; all resolved in PRD §17 and
the three most consequential re-verified against the code first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7iserver/src/models.ts — OpenRouter catalog with two capability mechanisms on purpose: a hand-maintained allowlist is authoritative for the three curated jobs (a wildcard rule would silently admit a future non-tool variant of a name we trust), while the advanced catalog is filtered by the live feed's supported_parameters (274 of 342 rows). Serves stale on a fetch failure rather than emptying the picker, and coalesces concurrent misses into one fetch. Curated chains are grounded in a live spike against openrouter.ai on 2026-07-27, not invented: every id is present and tool-capable, with real pricing recorded in the source. The spike also settled PRD open question B3 — the free tier's existing default qwen/qwen3.6-35b-a3b IS tool-capable, so no cost-affecting model change is needed. server/src/rate-limit.ts — split turn-starts (10/min, unchanged) from continuation steps (40/min). The old single 10/min limiter would have 429'd a user's second turn within a minute once one turn became 12 requests. server/src/step-token.ts — HMAC bound on a client-driven loop, key derived from KEY_ENCRYPTION_SECRET via HKDF with a distinct info label so a step-token leak cannot be replayed against credential encryption. Documents its own limit: not single-use, so it bounds a token rather than a turn; the real spend ceiling is the per-user OpenRouter key limit. Harness deps on AppDeps are optional, so the single-shot /api/agent surface and its tests keep working untouched. server tests 103 -> 153, typecheck clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
The plan gate found that neither client could actually speak tool calls, and no step owned adding it — the whole loop was planned on a transport that did not exist. server/src/openrouter.ts gains toolCompletion() alongside chatCompletion(). They deliberately do NOT share a result type: a pure tool-call response has content: null, which chatCompletion maps to a 502. That is correct for the single-shot JSON protocol and would have failed on the harness's first step. toolCompletion errors only when a message has neither content nor a usable call, and never sends response_format (providers reject it alongside tools). ModelMessage gains the tool role plus tool_calls/tool_call_id on both sides. Providers reject a tool_calls array with no matching tool message, so the pair is documented as inseparable and pinned by a test. normalizeToolCalls drops calls with no id: the answering tool message must reference the id, and providers emit partial entries on truncation. Ollama is cut from tool mode (blocker B2), enforced by the literal type LLMStepParams.provider: 'openrouter'. It forces format:'json' (conflicts with tools), returns tool_calls arguments as an object rather than the JSON string every other provider sends, and exposes no capability metadata at all — which makes the S7 pre-flight check unimplementable there. Ollama stays on the JSON protocol and the UI will say so rather than degrading silently. server 153 -> 169 tests; root vitest 108 -> 124; gleam 140 unchanged; tsc -b and server typecheck clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
agent.gleam gains the tool-loop state machine: step counter, activity trail, union of touched paths, pkg_dirty, pending calls, and the accumulated reply. New messages AgentStepReturned / AgentToolStarted / AgentToolFinished / AgentStepBudgetReached; new effects CallAgentStep / ExecuteTool / KillExec / InstallDependencies. Design change from the PRD: the step counter lives in State, not in Lifecycle.Running. It is turn-level bookkeeping rather than a distinct lifecycle, and keeping Running at two fields means every existing case over it still compiles — which is what lets this unit land with update.gleam byte-for- byte unchanged (verified: git diff on update.gleam is empty). The transcript deliberately does NOT live in Gleam. Gleam holds counts, names, statuses, summaries and paths; opaque provider JSON and raw tool bodies stay in agent.mjs. Modeling them here would put file contents and fetched page bodies into the app model, and the trail is model-visible on the next step. Invariants pinned by test: - MAX_TOOL_STEPS stops the loop rather than dispatching another step. - A stale request_id is ignored at all five new entry points. - Cancel/timeout/failure emit KillExec alongside AbortAgent — a wedged command that outlives the turn quietly degrades the container. - Timeout rests in Idle, not TimedOut, so the user can retry immediately. - touched_paths dedupe and keep first-write order. - pkg_dirty clears only on a *successful* install (blocker B4), so a failed npm install still gets the turn-end retry. - turn_summary never claims "checked it builds" when the check failed. One existing test updated: cancel effects now include KillExec, with a comment explaining why the contract changed. gleam 140 -> 164 tests; vitest 124; npm run build green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
The unit that makes the harness Build's rather than generic: exec is wc.spawn(),
not a rented microVM, so the agent can run `npx tsc --noEmit`, read real
diagnostics, fix them, and re-check before handing back.
src/agent-tools.ts holds fs_list / fs_read / fs_write / fs_batch_write / exec
behind two rules, each enforced in exactly one function so there is one place to
audit:
- Writes go through project.FileApplied, never wc.fs. project.files is the
source of truth and the container FS is a replica that isSyncableTextFile
silently filters; a direct write would leave the editor, ZIP export, publish,
autosave, and the next turn's context reading stale bytes with no error
anywhere. Attested by grep: agent-tools.ts mentions wc.fs.writeFile only in
the comment explaining why it must not call it.
- Results never echo content. A write returns `ok · 412 bytes`. Tool results are
re-sent on every later step, so echoing pays for the same bytes repeatedly.
fs_batch_write validates the entire batch before applying any of it: half a
refactor is worse than none, because the model's next step would reason about a
file set that never existed as a coherent whole.
exec bounding, with the boundary described honestly as reliability + visibility
rather than airtight security (the user's own jsh can already run anything):
argv form, allowlist npm/npx/node, npx narrowed to {tsc, vite} — bare
`npx <pkg>` runs arbitrary remote code with no visible artifact, making it
strictly more capable than the `node -e` denied beside it. `node <file>` stays
allowed because the file had to be written first, and a write is chip-visible.
Refusals teach ("the dev server is already running... run npm run build")
because a bare no invites a retry.
Two bugs found by writing the tests:
- exec claimed its concurrency slot only after `await spawn`, so two calls in
one step could both pass the guard and race two installs against
node_modules. The claim is now synchronous, with a deferred kill for an abort
landing mid-spawn.
- validateExec checked shell metacharacters before the node -e rule, so
`node -e 'console.log(1)'` was refused with the generic reason instead of the
one that tells the model what to do instead.
- flushContainerWrites initially cleared debounced editor writes instead of
performing them, which would have silently lost the user's keystrokes.
Trail copy reports outcomes, not statuses: "Checked the code — found problems",
"Built cleanly". A test asserts no summary ever leaks a tool name or a raw
command.
vitest 124 -> 205; gleam 164; tsc -b, npm run build, server typecheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7iPOST /api/agent/step — one step of the client-driven loop, stateless by design.
The browser drives (it owns the WebContainer, so fs_*/exec execute there) and
carries the transcript in the request body, so Build's anonymous/local-first
promise holds: the server sees project bytes transiently, exactly as /api/agent
already does, and persists nothing new.
Wire shape is tree + fullFiles, not the whole project (blocker B3). A 160k-char
project would otherwise be re-uploaded on all 12 steps and re-checked against
the 2MB cap each time. Pinned by a test that a 400-file project still produces
a valid step request.
Tool-mode prompt drops the JSON envelope from the header entirely rather than
contradicting it later. The header is the strongest signal in the prompt, and
leaving the old shape there would invite answers the legacy adapter silently
accepts — so the regression would never surface as an error.
Step-budget nudge is binding, not advisory: on the final step the server both
injects the nudge AND offers no tools. Ending a long turn with a usable answer
beats an error that discards everything already done.
Legacy {reply, patches} answers decode into one synthetic fs_batch_write, so
there stays exactly one write path. Reported as usedLegacyAdapter so its hit
rate can be measured and the adapter deleted once it reaches zero.
Parity guards extended (U6). The tool specs are now a second intentionally-
duplicated surface (client executes, server declares) and a disagreement would
show up as a stalled turn rather than an error, so it is guarded like the prompt
and the starter template:
- identical tool names and descriptions across both files
- MAX_CALLS_PER_STEP and MAX_TOOL_STEPS in step with the Gleam actor constants
- no web tool ever named in the client surface (SERVER_ONLY_RULE_MARKERS)
- no client tool marked approval-gated — the tripwire for "trust the sandbox"
- the JSON-mode Rules block extracts unambiguously (advisory A13)
Guard verified to bite: drifting one server description fails with a named,
actionable message, then passes again on restore.
server 169 -> 198 tests; root vitest 205 -> 212; gleam 164; tsc -b and server
typecheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7iRecords the real state: the harness is fully built and deliberately dark. Every unit's evidence, the live OpenRouter spike that settled open question B3, the boundary attestations that hold today (and the two deferred to U7a/U7b), and the three bugs found while building rather than inherited. Validation: gleam 164, vitest 212, server 198, both smokes at full count (25/25 and 14/14), tsc -b and server typecheck clean. Merge-base was 140/108/103, so +223 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
The switch is flipped. callAgent no longer makes one single-shot call — it opens a turn, records the context that does not change between steps, and takes step 0; the actor drives continuation through callAgentStep once every tool call for a step has reported back. Retirement is complete, not partial (blocker B4). agent.Patch, AgentRequestSucceeded, InstallIfNeeded(patches), apply_patches, and installIfNeeded are all GONE. There is now exactly one write path (project.FileApplied, via the fs tools) and exactly one install trigger (InstallDependencies, driven by pkg_dirty). Leaving either in place would have meant two paths that can both decide to npm install. One chat bubble and one build-log entry per TURN, not per step: the Build Story is a narrative of what was built, and a twelve-step turn is still one thing the founder asked for. Pinned by a test that runs read -> write -> exec -> answer and asserts a single bubble and a single story entry. Blocker B6 — the timeout path was dead code. Nothing dispatched AgentTimeoutReached and update_agent had no branch for it, so a timed-out turn would have rendered no bubble at all. The turn deadline now dispatches it and update_agent renders a bubble that says work already done was kept. The deadline is per TURN, not per request: before the harness those were the same number, under a 12-step loop they are not. Blocker B7 — ResetProject emitted a bare AbortAgent and left the lifecycle Running, unlike New and Open. Under the loop a late step response would still have matched its request id and kept driving tool calls into the project the user had just reset. It now routes through AgentRequestCanceled, and abortAgent clears the turn maps so a late reply finds no context to continue from. Advisory A10 resolves by construction: improve_selected_element emits the same agent.CallAgent effect, which is now the loop's entry point, so the element picker converted without a separate change. gleam 164 -> 169; vitest 212; server 198; npm run build, both smokes 25/25 and 14/14, tsc -b and server typecheck all clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
The trail is deliberately NOT a growing list of tool calls. While the agent
works it is a single line that changes — bolt, verb, outcome. When the turn ends
it collapses to one quiet summary ("2 steps · 1 file · checked it builds") that
expands into receipts. A twelve-row log would push the composer off a laptop and
read as CI output to a founder whose prompt forbids jargon and file paths.
Amber stays the working-state signature: the bolt while working, nothing at all
once idle. Step rows use charcoal rather than slate, because slate (#5B7FA6)
reads blue at that size and competed with Show/Hide — the row's only interactive
affordance.
Driving it in a real browser found two genuine bugs that no unit test would
have:
1. THE AGENT WAS BLIND TO THE PROJECT. publishProjectFiles only fired from
SaveCurrentProject/ScheduleSave, and autosave is gated on the container being
hydrated — so on a fresh boot the snapshot was empty and fs_list returned
"The project has no files." The agent would have rewritten files it could not
read. Fixed at three levels: seeded from the authoritative file set in
setLastFiles (boot/remount, the only path that carries files on a first run),
seeded on ProjectLoaded/Created/FilesUpdated, and upserted incrementally on
every WriteFileToContainer so it never again depends on an autosave.
The smoke now asserts "Listed 12 files" rather than an empty project.
2. dispatchAgentSucceeded still referenced the removed AgentRequestSucceeded and
Patch constructors, and agent.mjs used dispatchAgentStepReturned and
dispatchAgentTimeoutReached without importing them. All three were runtime-
only paths that vitest never executes — the timeout would have thrown instead
of ending the turn.
New scripts/smoke-agent-harness.mjs drives a real multi-step turn with the real
executors against real project state; only the model is stood in for. 17/17,
covering the file-snapshot invariant (S5), one-line-not-a-log, verbs-not-tool-
names, one bubble per turn, chips, no amber when idle, 375px composer survival,
and cancel leaving no turn state behind.
gleam 169 -> 171; vitest 212; smokes 25/25 + 14/14 + 17/17.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7iThe failure mode this avoids is a <select> of 300 OpenRouter slugs in a settings modal, which makes the user responsible for a decision they cannot evaluate. So the picker sits beside Send (choosing how hard to think is part of composing a request), names the JOB rather than the model, and never shows a model id anywhere: "Quick changes" / "Most work" / "Hard problems". The blurbs talk about the tradeoff the user actually feels — their monthly budget, not milliseconds — because that is the real decision against a $5 cap. Hidden entirely for Ollama: it has no tool mode and no catalog, so a picker there would promise a choice that does not exist. The job -> model mapping is a fourth intentionally-duplicated surface (Gleam offers, server validates), so it joins the guarded set: prompt-parity now asserts settings.gleam's job_model matches the first choice of every CURATED_CHAINS entry in server/src/models.ts, in order, and that no job label contains a model id. Verified to bite — drifting one id fails with a diff. Managed mode PUTs the chosen id to /api/me/model, where the server re-validates it against the live tool-capable catalog; a model that cannot call tools is refused there rather than silently breaking the harness three turns later. Smoke grew to 22/22 with the picker checks. gleam 171, vitest 215, server 198, smokes 25/25 + 14/14 + 22/22, tsc -b and server typecheck clean. This completes release 1 (U1-U9). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Marks U7a/U7b/U8/U9 pass with their evidence, promotes the two deferred boundary attestations to held, records the blind-agent bug and the three runtime-only reference errors, and replaces the not-wired residual with the two that actually remain: no live-model run yet, and BYOK-OpenRouter still on the JSON path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Completes the core tool set: web_search, web_fetch, and web_post. Managed mode only — they need the SSRF guard, a server-held search key, and an authenticated caller, none of which BYOK has. The parity test now pins that asymmetry in both directions: each web tool must be declared server-side and must NOT be declared client-side. Ported from the reference implementation and adapted: url-guard.ts keeps the method-class split, which is the non-obvious part. Network boundary names (localhost/.local/.internal/metadata) are blocked for every method. Our own trust surfaces (scoutos.live, hyper.io, openrouter.ai, Clerk, Render) are blocked for WRITES only — an uncredentialed redirect-free GET sees what any anonymous client sees, while a POST could publish or spend. Suffix matching is dot-bounded so "evilscoutos.live" is not treated as ours. safe-http.ts does the connect-time IP check inside the socket's own DNS lookup, which is the only place that closes the rebinding window. isPrivateIp is written out explicitly rather than pulled from a dependency, covering loopback, private, link-local (cloud metadata), CGNAT, TEST-NET, multicast, and the IPv4-mapped and NAT64 wrappers of all of them. No redirects, no ambient credentials, byte and time caps. Verified live: a fetch of localtest.me (which resolves to 127.0.0.1) is blocked at connect time, so the custom lookup does work on this runtime. injection-guard.ts is the single choke point every web result passes through — sanitize, scan, wrap. It WARNS rather than blocking, because heuristics have false positives and a page about prompt injection must stay readable. The tool-coercion pattern is extended with Build's own tool names. Delimiter neutralization is joiner-tolerant, so a forged "<<<END UNTRUSTED" split by zero-width joiners cannot fake an end-of-untrusted-data boundary. The truncation budget is computed from real assembled overhead so the END marker always survives the slice. The tainted-turn rule is a Build addition the reference does not have: once anything has been read from the web, that turn's web_post is withdrawn from the offered tools AND re-checked server-side before sending, so a client that "forgot" cannot unlock the write. Scoped honestly in the source — it closes the channel Build's own server would lend to an injected instruction, not exfiltration in general, since the WebContainer has its own egress. web_post is the only gated tool in the harness. It never reaches the client as a runnable call; it comes back as an approval request carrying the exact method, URL, and body, because a card that summarizes is a card that hides something. Search prefers Brave when keyed and requires an explicit opt-in for keyless DuckDuckGo: keyless runs from shared Render egress, so throttling would hit every user at once. Failure says "unavailable" rather than returning nothing — an empty result set reads as "the web has nothing" and sends the model off to invent an answer. Both new guards verified to bite: removing the tainted-turn withdrawal fails 2 tests; leaking a web tool into the client spec set fails 3. server 198 -> 295 tests (+97); root vitest 215 -> 220; gleam 171; smokes 25/25 + 14/14 + 22/22; tsc -b, npm run build, server typecheck clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
U11 returned an approval from the server but nothing rendered it — so a web_post would have been silently dropped: a step with no client calls reads as "the model answered" and closes the turn. This completes the path. The actor gains pending_approval, and a step with no calls PAUSES rather than finishing while one is up. Both outcomes answer the model: approve sends and feeds back the response, decline feeds back "the user declined" — a turn that told the model nothing would stall until the deadline. The blocked case is enforced in the actor, not just hidden in the UI: an approve message for a tainted-turn approval still routes to decline. Three independent layers now have to agree before anything leaves the sandbox — the card has no send button, the actor refuses the send, and the server re-checks the taint. The card shows the whole request: plain-language headline naming the host, method and full URL in mono, and the exact body in a scrollable block. Ink on white with the amber left rule, the same "needs your attention" language as the preview-error card — never a red alarm. While an approval is up the working line says "waiting for you" rather than "hyper is thinking", because it isn't. gleam 171 -> 177; smoke 22 -> 29 checks, including that the card shows the exact body rather than a summary and that a tainted turn offers no send button. vitest 220, server 295, all smokes green, tsc -b and typecheck clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
B1 — the web_post approval flow broke the turn at the provider, and did so AFTER the side effect. The server excluded the gated call from `toolCalls`, so the client echoed an empty assistant tool_calls array while sending a tool result for id p1. Providers reject a tool message with no matching call, so the user would have seen "Agent request failed (502)" after their data had already been POSTed. The server now also returns `transcriptCalls` — everything the model emitted — and the client echoes that. This is the invariant the code documents in three places and nothing tested; there are now three tests, including one asserting every tool_call_id in a post-approval step is announced in the same request. B2 — server-run web reads never reached the trail. AgentToolStarted updates an existing row and deliberately ignores unknown ids (a stale-id guard), but server steps have no row to update, so every web_search, every web_fetch, and the injection warning with them were silently discarded. Rather than weaken the stale-id guard, a distinct AgentServerStepRecorded appends a finished row. The PRD specified that row verbatim; it did not exist in the shipped product. B3 — BYOK regression. The single-shot path applied patches directly and never dispatched AgentToolFinished, which is the only thing that populates touched_paths and pkg_dirty. So BYOK turns had no narration chips, contributed no paths to the Build Story, and — worst — never triggered npm install: adding a dependency left the preview broken with an unresolved import. BYOK now adapts its response into the same synthetic fs_batch_write the managed legacy adapter uses, so both modes go through identical machinery. (My first attempt had an ordering bug of its own: AgentToolFinished emits CallAgentStep synchronously, which would have fired a second provider call — the turn is dropped first.) Advisories also fixed: - A4: web_search result sets and web_post response bodies now go through guardWebContent. Titles, snippets, and a POST target's reply are all attacker-authored, and the PRD said EVERY web-tool result. - A5: isPrivateIp gained fec0::/10 site-local, 2002::/16 6to4 (2002:7f00:1:: is 127.0.0.1 in a costume), and hex NAT64/IPv4-mapped forms. - A6: deny patterns were inconsistently case-sensitive, leaving .ENV, ID_RSA and .NPMRC writable. - A8: loadJob was never called, so the picker silently reset on every reload despite persistJob writing faithfully. SettingsLoaded now carries the job. - A3: docs/architecture.md still described the retired patch protocol; the Agent protocol section is rewritten and the validation battery updated. gleam 177 -> 180; server 295 -> 305; vitest 220; smokes 25/25 + 14/14 + 29/29. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
…eads) A1 — a step can return BOTH a client call and a web_post, which left two independent triggers for the next step: the last tool finishing, and the user resolving the approval. Two in-flight requests sharing one turn would race drainToolResults and clobber each other's stepToken and toolCalls. Fixed at both levels: the actor withholds CallAgentStep while an approval is pending, and runStep is serialized per turn with a queued-step slot so the later trigger is deferred rather than dropped — its results still reach the model next step. A2 — inline web reads were lost the moment the step ended. The server pushed the assistant/tool pair into its LOCAL message list, but step N+1 rebuilds from the request body via buildToolModeMessages, which knows nothing about it. So the agent would read a docs page, write a file, and on the next step have to fetch the same page again — on a turn whose taint flag had already cost it web_post. The server now returns the inline calls in transcriptCalls and their answers in serverToolResults, and the client echoes the pair. Writing the A2 test caught a bug in the A2 fix: the loop-local filter variable shadowed the accumulator of the same name, so every push landed in a local that was discarded each iteration and transcriptCalls came back empty. gleam 180 -> 182; server 305 -> 307; vitest 220; smokes 25/25 + 14/14 + 29/29. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
The counterweight to unattended writing. "Trust the sandbox" means a turn can rewrite six files without asking, so there has to be a way back — and until now the only recovery from a bad multi-file turn was "Reset to default app", which nukes the project. The snapshot rides the agent actor: AgentRequestStarted carries the project as it stands, captured before the turn writes anything. clear_turn deliberately does NOT drop it — undo has to outlive the turn that produced it — so it is replaced when the next turn starts and dropped on project navigation, which resets the whole actor. Only the most recent turn is undoable, and only when it actually changed something. A stack would need persistence and a UI; the realistic regret is always "that last one made it worse". An undo button after a read-only turn would promise to reverse nothing. Blocker B5 was specifically that the obvious mechanism does not work: project.RemountProject discards its files argument at the interpreter and remounts a JS-side cache that may be several turns stale, so restoring through it would look right and silently mount the wrong thing. RestoreSnapshot carries the files explicitly and goes out via dispatchWebContainerRemountRequested, which is the only remount path that passes them through. Verified end to end in a browser rather than by inspection: a turn clobbers src/main.tsx, undo is offered, clicking it restores the byte-exact pre-turn content, and the button is gone afterwards. gleam 182 -> 187; smoke 29 -> 34 checks; vitest 220; server 307; all three smokes green; tsc -b and typecheck clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Without it the agent physically cannot finish a refactor: it writes Card.tsx to replace OldCard.tsx and the old file lingers forever, eating the context budget every turn, showing in the Files list, and shipping to scoutos.live on the next publish. Until now the model had no way to express deletion at all. Bounded hard: one path per call, no directories, the global path policy applies, and the guarded set (entry module, database bridge, build config, BRAIN.md) refuses with an explanation rather than a bare no — a refusal that doesn't teach invites a retry. New actor surface: project.FileRemoved + DeleteFileFromContainer, and templates.remove_file as the counterpart to upsert_file (additive only, so the byte-for-byte starter-template drift test stays green). Deleting the open file moves selection to a sibling rather than leaving the editor pointing at nothing. Two bugs the end-to-end smoke caught that the unit tests could not: - FileRemoved was missing from update.gleam's auto-save branch, so a delete never scheduled a save and never republished the file list. - The snapshot removal was inside deleteFileFromContainer, AFTER `await bootGate` — so a delete during boot would leave the file in the agent's snapshot indefinitely and fs_list would keep offering a file that is gone. It now happens synchronously in the interpreter, exactly like the write path does with publish_project_file. Same class as the blind-agent bug: the snapshot must never depend on an async path settling. gleam 187 -> 191; vitest 220 -> 225; server 307; smoke 34 -> 37 checks (delete removes it, the entry file cannot be deleted). All twelve units are now done. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
BYOK now runs the same tool loop as managed mode, driven entirely from the browser against the user's own key. The step budget is enforced by the actor (there is no step token without a server), and no web tools are offered — they need the SSRF guard and a server-held search key, so their absence is structural rather than conditional. Ollama stays on the single-shot JSON protocol and its response is adapted into the same synthetic fs_batch_write, so every mode goes through identical machinery. src/agent.ts gains the tool-mode prompt as a mirror of server/src/prompt.ts, with SHARED_RULES lifted verbatim and WEB_TOOL_RULES deliberately absent. The parity test now compares the built tool-mode prompt across both files and asserts the web rules appear only when the server actually offers them — verified to bite by drifting one shared rule. Two bugs found by driving the loop against a stubbed provider: 1. FetchLLMClient stored `globalThis.fetch` unbound and invoked it as `this.fetchFn(...)`, so the browser saw `this` as the client instance and threw "Illegal invocation". PRE-EXISTING — it would have broken BYOK's JSON path in production too. Unit tests never caught it because injected test doubles are plain functions with no `this` to lose. 2. The transcript window was ONE STEP WIDE in both modes: each step replaced toolCalls/toolResults instead of accumulating, so the model had no memory of what it had already done and re-decided from the file tree every step. Measured at 12 steps for work that takes 2 — it burned the entire budget rediscovering the same thing, then hit the ceiling and answered "Done." Both modes now accumulate, which is also what makes the step budget mean what it says. Verified end to end against a stubbed OpenRouter: 2 steps, correct reply, correct narration chips, file written, undo offered. vitest 225 -> 234; gleam 191; server 307; smokes 25/25 + 14/14 + 37/37. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
… most
First run against a real model over the real network driving the real
WebContainer. It immediately exposed a defect that 500+ tests and three stubbed
smokes could not:
THE USER'S REQUEST WAS DROPPED AFTER STEP 0. `userPrompt` was sent only when
stepIndex === 0, so from step 1 onward the model had tool results and no
request. It read files, found nothing to act on, and answered "Looks like you
sent a blank message — what would you like to create?" The turn completed
"successfully" with zero files changed, so nothing in the harness registered a
failure. Present in BOTH transports.
The stubs missed it because a stub does not care what you ask it. This is the
whole argument for a live run as the acceptance gate.
Fixed by passing userPrompt on every step, with a source-level regression guard
(the externals layer is not executed by unit tests) verified to bite by
reintroducing the condition.
Also strengthened TOOL_MODE_WORKFLOW on both parity-guarded sides after the
first run showed the agent reading six files — index.html and main.tsx twice
each — and then stopping: reading is now explicitly preparation rather than an
answer, re-reading is called out as a wasted step, and the starter page is named
as a placeholder to replace rather than describe back.
Measured, same prompt ("Add a footer ... Built with hyper"):
anthropic/claude-sonnet-4.6 5 steps · 22s · 6 provider calls · 1 file
Read main.tsx → Wrote main.tsx → Checked the code — found problems
→ Read main.tsx → Built cleanly
qwen/qwen3.6-35b-a3b (free) 10 steps · 9 provider calls · 2 files
... → Tried to run a command it may not run ×2 → Checked the code
— found problems ×2 → Built cleanly
Both wrote real code, typechecked it, found real errors, fixed them, and
re-verified before replying — S1 satisfied by observation, not assumption. The
footer renders in the live preview. The free tier works, which settles the
open question about its default model. The exec allowlist refused two commands
and the teaching refusal let the model recover rather than stall.
scripts/live-agent-run.mjs reads the key from gitignored .env (un-prefixed, so
vite never bundles it), injects it at runtime as a BYOK user would, and redacts
defensively. Verified: zero key occurrences in the diff.
vitest 234 -> 236; gleam 191; server 307; smokes 25/25 + 14/14 + 37/37.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7iCo-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
…t exposed scripts/live-agent-suite.mjs drives escalating turns in ONE session so chat history, project state, and the container accumulate the way they do for a real user. Each turn strains a different part of the harness. Result on anthropic/claude-sonnet-4.6 — S1 verification rate 3/3: turn steps calls wall verified batched prev-err multi-file feature 6 5 68s yes yes 0 edit existing code 8 7 46s yes no 0 new dependency 9 10 68s yes no 0 It built a real working app across the three turns: "Taskwise", 16 files, a Header/TaskList/App split written with ONE fs_batch_write, working add-task and strike-through, and date-fns installed and used for relative timestamps. Every turn typechecked, found real errors, fixed them, and rebuilt before replying. Two findings worth more than the pass: - ADVISORY A6 DID NOT MATERIALIZE. Zero preview errors across all three turns, including the one that ran npm install. The concern was that multi-step write bursts would crash-loop the dev server and flood the Try-to-fix card; the flush-before-exec batching appears to be enough. Recorded as measured rather than assumed. - The pkg_dirty path works end to end: package.json changed, "Installed dependencies" ran once, and the build passed after it. Two prompt fixes the runs exposed, applied to both parity-guarded sides: - Replies were emitting markdown — "- **`Header.tsx`** — shows the orange..." rendered as literal asterisks and backticks in the chat bubble, which shows plain text. The workflow now says the final message is plain conversational text shown as-is, two or three sentences, in the user's terms not the codebase's. Confirmed fixed by a follow-up run. - The model burned a step guessing at commands the allowlist refuses. The workflow now enumerates exactly what exec runs, so it does not have to guess. Also fixed my own measurement bug: the suite counted installs from console output and missed real ones the trail had recorded. It reads the trail now — what the user actually sees is the honest source. Key handling unchanged: gitignored .env, un-prefixed, injected at runtime, redacted from output; screenshots land in gitignored scripts/.smoke. gleam 191; vitest 236; server 307; smokes 25/25 + 14/14 + 37/37. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
… files A second review pass over everything that landed after the last judge found four correctness defects, three of them reproduced against the compiled actor and the real Hono app. All four broke an invariant an earlier review had already forced a fix for. B1 — STALE SNAPSHOT SURVIVES A PROJECT SWITCH (data loss). The comment claimed the undo snapshot was "dropped on project navigation (which resets the whole actor)". It was not: navigation dispatches AgentRequestCanceled, and that was a no-op when the lifecycle is Idle — exactly the state you are in between turns. So a finished turn's card, its Undo button, and project A's file snapshot all survived opening project B. One click restored A's entire file set over B and marked it dirty for autosave. AgentRequestCanceled now clears the turn and the snapshot from Idle too. B2 — APPROVING WHILE CLIENT TOOLS RUN ORPHANS A CALL AND DOUBLE-RUNS THE STEP. The code comment described the guard; the guard did not exist. Approving a web_post while an exec from the same step was still running sent a transcript whose exec call had no answering result (provider 400 -> 502, turn dies), and then the exec finishing drove a SECOND step at the same index — duplicate writes, possibly a duplicate npm install. Step tokens are deliberately not single-use, so nothing server-side caught it. The actor now drives the next step from exactly one place: the approval when nothing else is pending, otherwise the last tool to finish. B3 — a second web_post in one step was announced and never answerable, because only approval.call_id ever receives a result. Extra gated calls are now refused inline with a legible reason. B4 — a client call emitted ALONGSIDE a server-run web tool was told "your result arrives next step" and then silently dropped, because the inner loop overwrites `result`. The turn could report done with a read still outstanding. Reachable as soon as web tools are offered, since the prompt tells the model to read before it writes. Deferred calls are now carried forward. New tests assert the shared invariant directly: every announced call must be answerable — either echoed for the browser to run, or already answered by the server. Re-validated live after the fixes (2 turns, Sonnet): both verified, including a 14-step turn that hit a missing tsconfig, fixed it, installed a dependency, and ended on "Checked the code — no problems". gleam 191 -> 194; server 307 -> 309; vitest 236; smokes 25/25 + 14/14 + 37/37. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
The provider resolution was built correctly but the key appeared in no env config — render.yaml or .env.example — so web_search would have shipped reporting itself unavailable. Declared as optional (sync: false) with a note on why keyless DuckDuckGo is not the fallback on a shared-egress service. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
twilson63
temporarily deployed
to
agent-harness - build PR #37
July 28, 2026 14:20 — with 
Render
Destroyed
hyperio-mc
approved these changes
Jul 28, 2026
hyperio-mc
left a comment
Collaborator
There was a problem hiding this comment.
Review Summary
This is a well-architected, thoroughly tested, and honestly documented PR. The design invariants are clear (one write path, one install trigger, trust the sandbox), the bugs found during development are fixed with regression tests, and the prompt-parity guards prevent silent drift between client and server.
What stands out
- Clean Gleam state machine — pure data, no mocks needed, excellent test coverage (194 Gleam + 236 vitest + 309 server + 37 new smoke tests)
- Honest bug documentation — real bugs found only by live runs, each with a regression test
- Prompt parity tests guard against client/server drift
- Undo is well-scoped — snapshot travels with the effect, not through RemountProject
- Job picker UX (Quick/Standard/Hard instead of model ids) is thoughtful
Before merge
- Get CI running on the branch to verify test counts
- Track the managed-mode flip (
VITE_MANAGED_AUTH) separately — acknowledged in PR body
Minor suggestions (non-blocking)
- Consider a dev-only assertion if
runStepis called whileinFlightis true without a queued step - Document the
globalThis.__buildProjectFilesupdate order in a single place
Overall: looks good to merge once CI confirms tests pass. The managed-mode flip should be a separate, tracked change.
🤖 Reviewed by MC Agent
twilson63 added a commit
that referenced
this pull request
Jul 28, 2026
PR #37 merged to main — which Render auto-deploys — with a local test run as the only thing between the branch and production. That worked because the run happened; it is not a control. Three jobs mirroring the real battery: app (gleam test + vitest + tsc + build), server (vitest + typecheck, neither of which the app job covers), and smoke, which drives all three browser smokes including the agent harness one — the test that catches desync between project.files and what the agent can see. Gleam is pinned to the same version render.yaml installs, so a build that passes here is the build that ships. The dev server matches production's VITE_MANAGED_AUTH=false. Also declared playwright as a devDependency. The smokes have been running on a stray local install for months — CI would not have had it, and neither would a fresh clone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces Build's single-shot
{reply, patches[]}agent with a boundedtool-calling loop that uses the StackBlitz WebContainer as the agent's control
plane.
The decisive asymmetry: HyperChat rents an E2B microVM for
run_code. Buildalready owns a sandbox, so
execiswc.spawn()— the agent can runnpx tsc --noEmit, read real diagnostics, fix itself, and only then hand back.What it does now, live
Real model, real network, real container:
Across a 3-turn session it built a working 16-file task app: a
Header/TaskList/App split written with one
fs_batch_write, add-task andstrike-through interactions, and
date-fnsinstalled via thepkg_dirtypath.S1 verification rate 4/4 — every turn typechecked, found real errors, fixed
them, and rebuilt before replying.
Tools
fs_list/fs_read/fs_write/fs_batch_write/fs_deleteexecwc.spawn()web_search/web_fetchweb_postPlus a curated job picker (Quick changes / Most work / Hard problems — never a
model id), an activity trail, and per-turn undo.
Load-bearing invariants
project.FileApplied.project.filesis the source of truth; the container FS is a replica thatisSyncableTextFilefilters. A directwc.fs.writeFiledesyncs the editor,ZIP export, publish, autosave, and the next turn's context — silently.
InstallDependencies, driven bypkg_dirty, clearedonly on a successful install.
fs_*andexecrun unattended; onlyweb_postgates,because it is the one tool that reaches out of the WebContainer.
web_postfor the rest of it.server holds users' source code.
Review history
A plan gate returned NOT APPROVED with 12 blockers before any code was
written (the tool transport did not exist;
openrouter.ts502'd on the exactshape of a tool-call response; the timeout path was dead code;
ResetProjectbypassed cancel). An implementation judge then found 3 more, and a second
review found 4 more — including one where undo could restore project A's
files over project B. All resolved, each with a regression test.
Bugs that only a real run could find, all fixed:
like you sent a blank message" while the turn reported success with zero
files changed.
tree every step — 12 steps for work that takes 2.
FetchLLMClientheldglobalThis.fetchunbound (pre-existing; would havebroken BYOK's JSON path in production).
autosave, which is gated on the container being hydrated.
Validation
Merge-base was gleam 140 / vitest 108 / server 103.
Previews set
VITE_MANAGED_AUTH: "true"but rewrite/api/*to theproduction API, and the API service has previews disabled (the disk pins it
to one instance). So the preview runs managed mode against an API with no
/api/agent/stepand every turn 404s. Ignore it.Production is
VITE_MANAGED_AUTH: "false", so merging exercises the BYOKloop — the path validated live here — and the API changes are purely additive.
Not in this PR
BYOK. Both transports share the actor, executors, prompt and accumulation, and
every fix landed in both — but
/api/agent/stephas only seen fixtures.Needs a Clerk instance. Flip
VITE_MANAGED_AUTHseparately, not with thismerge.
web_*are managed-only, so they stay inert in prod until that flip.My recommendation against the workflow canvas is in there, with the schema
designed anyway so it can be overruled without re-deriving.
Full detail:
docs/agent-harness-prd.htmlanddocs/agent-harness-progress.html.🤖 Generated with Claude Code
https://claude.ai/code/session_019gS8yjMedGgujEad6zRD7i