Uh oh!
There was an error while loading. Please reload this page.
feat(headless): @maka/headless — single headless agent entry (eval mode), #31 - #42
Merged
Conversation
#31) First end-to-end slice of the agent experiment lab: run one Config against one Task in an isolated sandbox, capture the trajectory, score it, and return a ResultRecord. De-risks the integration (headless SessionManager run + sandbox + evaluator composing into one real run); matrix/compare/CLI and real-model backends are later additions. - contracts.ts: Task (instruction + fixture workspace + verification command), Config (backend/connection/model), ResultRecord. Minimal; systemPrompt/Execution/SWE-bench-pack ingestion deferred. - sandbox.ts: prepareWorkspace copies the fixture to a throwaway dir so the agent never mutates the source (isolation, not asking). - evaluator.ts: runVerification runs the Task's test command in the throwaway workspace after the agent finishes (config can't grade itself), with output cap + timeout kill. - runner.ts: runExperiment wires a pure-Node SessionManager (injected registerBackends keeps model/credential wiring out of the lab core), drives one turn, captures the InvocationResult trajectory via runtimeInvocationObserver, scores it, returns a ResultRecord. - tests: FakeBackend e2e (pass + fail fixtures, trajectory persisted as runtime-events.jsonl), sandbox isolation, evaluator pass/fail/timeout. Independent of the credential migration (#32/#33): the skeleton runs on FakeBackend and needs no credentials.
Completes the MVP loop on top of the walking skeleton: run a grid of Configs × Tasks, persist canonical results, and compare. - matrix.ts: runMatrix runs the full cross product (sequential; a thrown run becomes a failed cell instead of aborting the grid) with a per-cell onResult callback. - results.ts: ResultRecord JSONL is canonical truth; toComparisonTable derives a git-diffable markdown grid (tasks × configs, ✅/❌/⚠️ , pass-rate footer). ResultRecord gains an optional `error`. - backends.ts: the two concrete backend wirings, kept out of the engine. registerFakeBackend (deterministic). registerAiSdkBackend resolves a Config's slug against spec connections, reads the API key from a named env var (no secrets at rest), and wires a minimal AiSdkBackend (model + builtin tools + execute-mode permission). Telemetry/artifact/synthesis hooks omitted — a benchmark scores via the verification command. - runner.ts: the drain loop now auto-approves permission requests — a headless benchmark has no human to confirm; throwaway-workspace isolation is the safety net. - cli.ts: `maka-lab run <spec.json> [--out <dir>]` and `maka-lab compare <results.jsonl>`; task fixtures resolve relative to the spec. Exposed as the `maka-lab` bin. - tests (15 total): CLI end-to-end smoke (spawns the built bin on a fake spec → results.jsonl + comparison.md → compare), matrix cross-product + failed-cell, JSONL round-trip, table rendering. The real ai-sdk backend is typecheck-verified but not unit-tested (a live model call is non-deterministic and costs money); it needs a live smoke test with a real API key. Everything else is deterministic and green.
README documents the Config × Task model, the CLI (run/compare), the spec shape (incl. a real ai-sdk connection with apiKeyEnv), the two backends, and what's deliberately out of MVP scope. examples/demo.spec.json + examples/demo/marker.txt run green on the fake backend with no API key: maka-lab run examples/demo.spec.json --out /tmp/maka-lab-demo
examples/fix-add — a buggy add() with a failing node:test. A real model run must read the files, fix src.mjs, and turn `node --test` green. Live smoke confirmed end-to-end on DeepSeek (deepseek-chat): completed, passed, exit 0, 124-event trajectory using Edit/Write/Bash; the source fixture stayed buggy (the agent only touched the throwaway copy), proving sandbox isolation + headless permission auto-approve work against a real backend. README points at it as the canonical real-run example.
…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.
The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.
The package is the single headless agent entry point, not just a benchmark lab; eval is one mode of it. Pure rename — directory, package name, bin (maka-lab → maka-headless), root workspaces, and internal references. No behavior change; all 24 tests pass.
… purpose Addresses an external review of the eval harness. The fix is an explicit trust posture, never implicit host access: - Fail closed: a model-backed backend (ai-sdk / pi-agent) is refused before a run starts — it would execute shell / network / file tools on the host, and the throwaway workspace is a copy, not a sandbox. Only the inert FakeBackend runs in-process. Real-model eval lands with the isolated executor (follow-up). - Required grading boundary: TaskVerification.protectedPaths is no longer optional, so a config can't silently grade itself; the CLI rejects a spec whose task omits it. - Honest exit codes: infrastructure failures (invalid spec, refused backend, a run that crashed before producing a result) exit non-zero; a run that completed and merely failed its verification stays exit 0 as valid data. - Drop allowDangerousTools (it left shell_unsafe / network_send open) and deny every in-process permission request — nothing inert needs one. - CLI `run` → `eval`: encode the purpose at the command boundary, so eval can never be handed host trust via a flag or spec field. - Curate the public export surface; stop calling the throwaway workspace a "sandbox" in docs. 28 tests pass.
… the engine, drop dead wiring - P2: a run whose backend reports failure without throwing now writes invocation.failure into ResultRecord.error, so the comparison table (⚠️ ) and the CLI exit code agree it was not a trustworthy run — no more⚠️ -but-exit-0 that automation reads as success. - P3: move the grading-boundary check into the engine. validateTaskVerification runs at the top of runExperiment, before any workspace / session / backend, so a direct (non-CLI) caller that omits protectedPaths fails fast; the CLI reuses the same function instead of a private duplicate. - P3: remove the dead registerAiSdkBackend / LabConnection wiring and its public export. The real backend is refused this build anyway; its wiring returns with the isolated executor (follow-up PR). Public API is now the fake-eval surface. 30 tests pass.
…l API
Round-3 review (all non-blocking):
- registerBackends is now optional; runExperiment defaults to the inert
FakeBackend, the only backend this build runs. Minimal usage is
`runExperiment(config, task, { storageRoot })`; the CLI no longer passes a
backend, and registerFakeBackend leaves the public index (backends.ts stays
internal — the seam where real backends rejoin with the executor).
- Remove examples/fix-add (a real-model ai-sdk spec this build always refuses)
and its README mention, so every shipped example actually runs. The
real-backend example returns with the executor PR; refusal stays covered by
the runner / CLI tests.
(mergeable=false was a transient post-push GitHub state; PR #42 reports
MERGEABLE / CLEAN, no rebase needed.)
31 tests pass.jackwener pushed a commit
that referenced
this pull request
Jun 21, 2026
…l API
Round-3 review (all non-blocking):
- registerBackends is now optional; runExperiment defaults to the inert
FakeBackend, the only backend this build runs. Minimal usage is
`runExperiment(config, task, { storageRoot })`; the CLI no longer passes a
backend, and registerFakeBackend leaves the public index (backends.ts stays
internal — the seam where real backends rejoin with the executor).
- Remove examples/fix-add (a real-model ai-sdk spec this build always refuses)
and its README mention, so every shipped example actually runs. The
real-backend example returns with the executor PR; refusal stays covered by
the runner / CLI tests.
(mergeable=false was a transient post-push GitHub state; PR #42 reports
MERGEABLE / CLEAN, no rebase needed.)
31 tests pass.jackwener pushed a commit
that referenced
this pull request
Jun 21, 2026
feat(headless): @maka/headless — single headless agent entry (eval mode), #31
jackwener pushed a commit
that referenced
this pull request
Jun 21, 2026
feat(headless): @maka/headless — single headless agent entry (eval mode), #31
jackwener pushed a commit
that referenced
this pull request
Jun 21, 2026
…l API
Round-3 review (all non-blocking):
- registerBackends is now optional; runExperiment defaults to the inert
FakeBackend, the only backend this build runs. Minimal usage is
`runExperiment(config, task, { storageRoot })`; the CLI no longer passes a
backend, and registerFakeBackend leaves the public index (backends.ts stays
internal — the seam where real backends rejoin with the executor).
- Remove examples/fix-add (a real-model ai-sdk spec this build always refuses)
and its README mention, so every shipped example actually runs. The
real-backend example returns with the executor PR; refusal stays covered by
the runner / CLI tests.
(mergeable=false was a transient post-push GitHub state; PR #42 reports
MERGEABLE / CLEAN, no rebase needed.)
31 tests pass.jackwener pushed a commit
that referenced
this pull request
Jun 21, 2026
feat(headless): @maka/headless — single headless agent entry (eval mode), #31
jackwener pushed a commit
that referenced
this pull request
Jun 21, 2026
…l API
Round-3 review (all non-blocking):
- registerBackends is now optional; runExperiment defaults to the inert
FakeBackend, the only backend this build runs. Minimal usage is
`runExperiment(config, task, { storageRoot })`; the CLI no longer passes a
backend, and registerFakeBackend leaves the public index (backends.ts stays
internal — the seam where real backends rejoin with the executor).
- Remove examples/fix-add (a real-model ai-sdk spec this build always refuses)
and its README mention, so every shipped example actually runs. The
real-backend example returns with the executor PR; refusal stays covered by
the runner / CLI tests.
(mergeable=false was a transient post-push GitHub state; PR #42 reports
MERGEABLE / CLEAN, no rebase needed.)
31 tests pass.jackwener pushed a commit
that referenced
this pull request
Jun 21, 2026
feat(headless): @maka/headless — single headless agent entry (eval mode), #31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #36 (auto-closed when its branch
claude/labwas renamed toclaude/headless; #36 had 0 comments / 0 reviews, nothing lost).What
@maka/headless(was@maka/lab) is the single headless agent entry point — not just a benchmark lab. eval (Config × Task → score) is one mode of it; an operational mode can slot into the same entry later (not built here, YAGNI).This PR
@maka/lab→@maka/headless(dir, package name, binmaka-headless, root workspaces). No behavior change; 24 tests pass.protectedPathsrequired (grading boundary is a conscious choice, not an omission)Follow-up PR
Container executor (Docker shell, env allowlist so Bash can't inherit host env, workspace mount, network policy) — only then does real-model eval run.
Refs #31.