feat(lab): @maka/lab — headless agent experiment lab (MVP, #31) by Astro-Han · Pull Request #36 · apache/maka

Astro-Han · 2026-06-17T05:31:08Z

Implements the MVP from #31. Independent of #32 / #33 — the lab runs on the FakeBackend with no credentials, and the real backend reads keys from an env var, not the credential store.

What & why

A headless lab for measuring an agent configuration: run a Config × Task grid in isolated sandboxes, capture each trajectory, score it with the task's own test command, and compare. Vary the model/connection (Config), hold the work (Task), read pass/fail off the table.

Config × Task → throwaway workspace → headless agent run → trajectory
↓
ResultRecord (JSONL) ← verification command

What's in this PR

New package @maka/lab (added to the workspaces list):

contracts — Task (instruction + fixture workspace + verification command), Config (backend / connection / model), ResultRecord (one canonical row per run). Minimal on purpose; kept in @maka/lab for now, promotable to @maka/core once a second consumer exists.
sandbox — copies a task fixture into a throwaway dir; the source is never mutated and runs never bleed into each other. Isolation, not asking, is the safety model.
evaluator — runs the task's verification command (exit 0 = pass) in the throwaway workspace after the agent finishes, so a config can't grade itself; output-capped + timeout-killed.
runner — drives one headless turn through SessionManager, captures the trajectory via runtimeInvocationObserver, and auto-approves permission prompts (a benchmark has no human to confirm; the throwaway workspace is the boundary).
matrix — the full Config × Task cross product; a thrown run becomes a failed cell instead of aborting the grid.
results — ResultRecord JSONL is canonical truth; a git-diffable markdown comparison table is derived from it.
backends — the two concrete wirings, kept out of the engine: fake (deterministic) and ai-sdk (real model; key from a named env var, minimal AiSdkBackend = model + builtin tools + execute-mode permission).
CLI — maka-lab run <spec.json> [--out <dir>] and maka-lab compare <results.jsonl>.
examples — demo.spec.json (fake, no key) and fix-add.spec.json (a real coding task).

Live validation

The real ai-sdk backend was smoke-tested end-to-end on DeepSeek (deepseek-chat) with examples/fix-add (a buggy add() + a failing node:test):

completed, passed, exit 0, a 124-event trajectory using Edit / Write / Bash
the source fixture stayed buggy — the agent only edited the throwaway copy, proving sandbox isolation + headless permission auto-approve work against a real model

Testing

15 deterministic tests (FakeBackend), all green: CLI end-to-end smoke (spawns the built bin → results.jsonl + comparison.md → compare), matrix cross-product + failed-cell, JSONL round-trip, table rendering, sandbox isolation, evaluator pass/fail/timeout, walking-skeleton e2e. Plus the live DeepSeek smoke above.

Out of scope (deliberate — pure additions later)

Parallel matrix execution, LLM/rule evaluators (today: command exit code), Docker / network isolation, SWE-bench pack ingestion, a richer report than the markdown grid, and promoting the contracts into @maka/core.

#31) First end-to-end slice of the agent experiment lab: run one Config against one Task in an isolated sandbox, capture the trajectory, score it, and return a ResultRecord. De-risks the integration (headless SessionManager run + sandbox + evaluator composing into one real run); matrix/compare/CLI and real-model backends are later additions. - contracts.ts: Task (instruction + fixture workspace + verification command), Config (backend/connection/model), ResultRecord. Minimal; systemPrompt/Execution/SWE-bench-pack ingestion deferred. - sandbox.ts: prepareWorkspace copies the fixture to a throwaway dir so the agent never mutates the source (isolation, not asking). - evaluator.ts: runVerification runs the Task's test command in the throwaway workspace after the agent finishes (config can't grade itself), with output cap + timeout kill. - runner.ts: runExperiment wires a pure-Node SessionManager (injected registerBackends keeps model/credential wiring out of the lab core), drives one turn, captures the InvocationResult trajectory via runtimeInvocationObserver, scores it, returns a ResultRecord. - tests: FakeBackend e2e (pass + fail fixtures, trajectory persisted as runtime-events.jsonl), sandbox isolation, evaluator pass/fail/timeout. Independent of the credential migration (#32/#33): the skeleton runs on FakeBackend and needs no credentials.

Completes the MVP loop on top of the walking skeleton: run a grid of Configs × Tasks, persist canonical results, and compare. - matrix.ts: runMatrix runs the full cross product (sequential; a thrown run becomes a failed cell instead of aborting the grid) with a per-cell onResult callback. - results.ts: ResultRecord JSONL is canonical truth; toComparisonTable derives a git-diffable markdown grid (tasks × configs, ✅/❌/⚠️, pass-rate footer). ResultRecord gains an optional `error`. - backends.ts: the two concrete backend wirings, kept out of the engine. registerFakeBackend (deterministic). registerAiSdkBackend resolves a Config's slug against spec connections, reads the API key from a named env var (no secrets at rest), and wires a minimal AiSdkBackend (model + builtin tools + execute-mode permission). Telemetry/artifact/synthesis hooks omitted — a benchmark scores via the verification command. - runner.ts: the drain loop now auto-approves permission requests — a headless benchmark has no human to confirm; throwaway-workspace isolation is the safety net. - cli.ts: `maka-lab run <spec.json> [--out <dir>]` and `maka-lab compare <results.jsonl>`; task fixtures resolve relative to the spec. Exposed as the `maka-lab` bin. - tests (15 total): CLI end-to-end smoke (spawns the built bin on a fake spec → results.jsonl + comparison.md → compare), matrix cross-product + failed-cell, JSONL round-trip, table rendering. The real ai-sdk backend is typecheck-verified but not unit-tested (a live model call is non-deterministic and costs money); it needs a live smoke test with a real API key. Everything else is deterministic and green.

README documents the Config × Task model, the CLI (run/compare), the spec shape (incl. a real ai-sdk connection with apiKeyEnv), the two backends, and what's deliberately out of MVP scope. examples/demo.spec.json + examples/demo/marker.txt run green on the fake backend with no API key: maka-lab run examples/demo.spec.json --out /tmp/maka-lab-demo

examples/fix-add — a buggy add() with a failing node:test. A real model run must read the files, fix src.mjs, and turn `node --test` green. Live smoke confirmed end-to-end on DeepSeek (deepseek-chat): completed, passed, exit 0, 124-event trajectory using Edit/Write/Bash; the source fixture stayed buggy (the agent only touched the throwaway copy), proving sandbox isolation + headless permission auto-approve work against a real backend. README points at it as the canonical real-run example.

…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as ⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.

The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.

…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as ⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.

The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.

…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as ⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.

The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.

…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as ⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.

The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.

Astro-Han · 2026-06-17T05:31:08Z