Uh oh!
There was an error while loading. Please reload this page.
feat(matrix): author a policy, then evaluate the program - #36
Merged
Conversation
The README stopped at the single-benchmark API. It said nothing about the transport axis, the streaming sandbox, the pace, the authoring phase, or how several games are pooled — every one of which changes what a number means. Add them, each with the measurement that justifies it: per-decision spawning scores 4 where the same program scores 1948 under a persistent process; an unpaced asynchronous game completes twelve decisions in 2 ms and so measures the host rather than the player; and across nine cells of the first study the rank correlation between actions delivered and score was 0.94, which is the reason `author=` exists. Add `examples/agent-runtime-matrix.mts`. `harness=<launcher>` spawns a program, so that axis is one vendor wide. `RunCellOptions.driver` is the seam that makes it many: one runtime backend per profile, and claude-code, codex and opencode sit in one table on one game, one clock and one sensor. The example reuses `runMatrix` and `createTangleRuntimeDriver` rather than restating either, and it names its own limit — a per-decision runtime call cannot keep state, so it measures delivery unless the profile authors a policy instead.
… section The first code block in the README did not run. It referenced `target`, `chooseAction` and `privateKey`, none of which it declared, and used `satisfies` on an undefined name. A reader who copied it got a type error before they got a benchmark. It now imports the bundled 2048 target, needs no ROM, and was executed to confirm the output comment: it verifies `first-legal-move` and `first-merge`. The matrix section I added last read like it was written to impress rather than to inform. Rewritten to the repo's own standard: no em-dashes, no capitalised words mid-sentence, no "this is not X, it is Y", no heading that argues with itself, and no sentence over 25 words. Section headings are now nouns: Transports, Streaming, Authoring, Several games, Many harnesses. The three declarations a streamed protocol needs are a list instead of a paragraph. Every measurement stays.
It was 1042 lines across 17 top-level sections. A reader met install, one example, then twenty-five sections of calibration and adapter theory before learning that an arena existed. Reference material and onboarding were the same document, so neither worked. The README is now 231 lines: what it is, install, one example that runs, the one-method agent interface, how to compare agents, when an episode ends, how to onboard a game, and a table of links. 700 lines moved into docs/ as reference pages: arena, calibration, observation, verification, episodes, long-horizon. Four adapter sections that existed only in the README - native process, native desktop, PyBoy, Steam and Xbox - moved into docs/adapters.md, which already held the other four. The driver section was already covered there in full and became a pointer. Nothing was deleted. Every heading that existed before this change is still present, every link resolves, and the boundary check passes. Front-page prose now holds no em-dashes, no capitalised words mid-sentence, and no sentence over 25 words. The deeper reference pages still carry both and are a separate pass.
…runs Two defects found by running the authoring phase for real. A protocol may state `pace=0`, which is correct for a scored episode played by a program. Practice is not that. The practice horizon was computed as the build budget divided by the pace, so `pace=0` produced 480,000 turns of an unpaced game. Measured on the first run: 449,821 observation files and 1.7 GB of disk in one cell. Practice now has a pace floor of 250 ms and a hard cap of 20,000 turns, which does not depend on any declared number. The runner printed one line per COMPLETED cell. A cell that takes hours therefore prints nothing for hours, and a slow run cannot be told from a hung one without attaching to the process. Measured: an authoring cell spent 2h13m playing 2000 decisions with a compiled search and reported nothing throughout, and its status was read as progress when no progress had been checked. A cell is now announced before it runs, states its build budget when it has one, and prints an elapsed heartbeat every minute while it is still going.
`author=` gives an agent one shot. It builds blind, is scored, and never sees its own score, which is a single sample rather than a climb. The loop already exists in `@tangle-network/agent-runtime`: `improve` with `surface: 'code'` forks a worktree per candidate, runs a bounded number of shots, meters each against the exact profile, and feeds the verifier's feedback into the next shot. Writing a second loop here would put the two out of step and measure whichever one drifted. What that loop cannot know is whether a program plays a game well. This is that function. It runs the candidate's program against a fresh scored instance, attests the replay, and reports the score with the milestones it reached and the ones it did not. A score is not a pass. A test suite is binary and there is nothing left to win once it is green; a game score always has a higher number above it. So a candidate that fails to START is `ok: false` with the stdio protocol restated, which the next shot can act on, and a candidate that plays badly is `ok: true` with a low number, which is a different finding. `keepGoing` says the program works and another shot is still worth spending. The author is told its own result and nothing else. Naming the contract, the reference or the evidence channel would hand it the grader, and a program written against the grader is not a program that plays the game. `watchEvidence` and `channel` are exported from matrix-run rather than copied, so the score here and the score in a matrix row are read by one path. Verified on a real candidate: a vocabulary cycler scored 1212 of 2048 over 120 decisions with a clean replay, and the feedback names the seven milestones it reached.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem this fixes
Under every transport playproof had, the agent plays live. Its start-up time and typing rate are therefore charged against its score.
Measured across nine cells of the first three-profile study: the rank correlation between ACTIONS DELIVERED and SCORE was 0.94. The score was very nearly a readout of how many moves reached the file. One agent in that study wrote a working expectimax searcher with self-play-tuned weights and scored 240, because the harness was grading how fast its author could type.
That is the wrong question for an arena that exists to compare what agent profiles can build.
author=A profile may now be asked to build a player instead of playing one.
Build. The agent gets a practice game streamed into its sandbox for
buildMinminutes. Nobody scores that game, and it runs at a different seed — practising on the game you are graded on is memorisation. It may play it, reimplement it, fit weights or train against it. It must leave an executablepolicybehind.Evaluate. That program runs cold against a fresh scored instance, in its own process, and the replay is attested by
attestRunexactly as any other. The agent is not running.Any solution qualifies, because the evaluation only ever sees a program answering decisions over stdio: a heuristic table, a search, weights fitted offline, a learned policy. Build cost lands in a
buildcolumn (usd,tokens,minutes) and never touches the play score.Two refusals the grammar enforces:
author=withtransport=stream(streaming an authored policy puts the author's clock back in), andbuildMinwithoutauthor(a budget for nothing). An agent that leaves no policy is blocked, not scored zero — building nothing and playing badly are different findings.It works
First real cell, opus, 8 minutes, $1.93, 4.32M tokens: a 408-line C program, self-compiling through a
shwrapper with agccfallback. 64-bit bitboard, four 65,536-entry precomputed row-transformation tables, expectimax over chance nodes weighted 0.9/0.1, probability-threshold pruning, and a generation-tagged transposition table with 4-way probing.The same model under the live transport produced Python that scored 240.
examples/agent-runtime-matrix.mtsharness=<launcher>spawns a program, so that column is one vendor wide.RunCellOptions.driveris the seam that makes it many: one runtime backend per profile, so claude-code, codex and opencode sit in one table on one game, one clock and one sensor. The example reusesrunMatrixandcreateTangleRuntimeDriverrather than restating either, and names its own limit inline.README
1042 lines to 231. Reference and onboarding were the same document, so neither worked: a reader met install, one example, then twenty-five sections of calibration theory before learning an arena existed.
700 lines moved into
docs/as reference pages (arena, calibration, observation, verification, episodes, long-horizon). Four adapter sections that existed only in the README — native process, native desktop, PyBoy, Steam and Xbox — merged intodocs/adapters.md, which already held the other four.The first code block did not run. It referenced
target,chooseActionandprivateKey, none of which it declared. It now uses the bundled 2048 target, needs no ROM, and was executed to confirm its output comment.Nothing was deleted. Every heading present before this change is still present, and every link in
README.mdanddocs/resolves.Checks