Uh oh!
There was an error while loading. Please reload this page.
chore(deps): agent-runtime 0.175.0, add agent-eval, and prove the join compiles - #37
Merged
Conversation
…n compiles The runtime was pinned at 0.142.3 against a current 0.175.0, and agent-eval was not installed at all, so the optimizer this study wants to use could not be typechecked against. 0.175.0 carries what a game needs. `VerifyResult` gained `keepGoing` and `score`, which is the disposition a score requires: a test suite is green and there is nothing left to win, while a game always has a higher number above it. `hillclimb.ts` already returns that exact shape. `examples/agent-eval-fit.mts` is the proof, and it exists to be compiled. A playproof cell types as an agent-eval `Scenario`, where `seedGroup` is what makes two profiles on one game a paired comparison instead of two unrelated samples. Playing a candidate types as a `Verifier`. A change on either side that breaks the join now fails the typecheck instead of failing a study three hours in. Also fixes a flaky test the bump exposed. It spawned a child that exits, slept 120 ms, then asserted the driver had already been told the child was gone. That asserts how busy the machine is. It now waits for the condition, and passed 10 of 10 consecutive runs.
…he statistics `compareOptimizationMethods` owns everything hard about comparing agent profiles: disjoint train, selection and final-test scenarios, bootstrap resamples, Bonferroni-adjusted simultaneous confidence across every contrast, a shared cost ceiling, and a policy that refuses to score a final test when the search history is incomplete. That machinery is the reason to route a game through it instead of tabulating means. On this repo's own three-profile study the ranking read cleanly as opus > sonnet > haiku; a paired bootstrap over the same replicates put two of the three contrasts across zero, and only opus over haiku survived. A table of means said one thing and an interval said another. Playproof supplies two things. A cell becomes a scenario, where `seedGroup` hands two surfaces the same seed for a replicate and makes the comparison paired rather than two unrelated samples of a noisy game. A played cell becomes a judge score, deterministic, because the game already produced the number. A blocked cell throws instead of scoring zero. A cell that could not be built did not play badly, and folding it into a zero would let an unbuildable clock read as a bad candidate. The surface reaches the game only through the profile, so none of this needs to know which of the ten searchable surfaces is being moved.
A cost of null could mean two different things and printed the same either way: the agent reported nothing, or the agent bills a plan and has no per-request figure to report. A study that reads those the same ranks the arm nobody could meter first on cost per point. Reuse the vocabulary the rest of the stack already has rather than inventing a second one. `authMode: 'api-key' | 'oauth'` is the union in `@tangle-network/agent-interface`, and the CLI agent registry passes the same value to a spawned backend as `CLAUDE_CODE_AUTH_MODE`. Both names are used here, so a run in this harness and a run in a sandbox describe themselves identically. The launcher honours an explicit `CLAUDE_CODE_AUTH_MODE`, and otherwise reads `api-key` when `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` is set and `oauth` when neither is. Verified on all three paths. The value travels with the meter to the row, and the runner prints an absent cost as its reason: `unbilled(oauth)` rather than a bare blank, and never `$0.0000`.
…were not An earlier count said 41 em-dashes across docs/. Most were markdown table cells using a dash as a value separator, and one is verbatim CI output inside a fence. Rewriting those would have damaged a table and falsified a quote. 20 were prose, and each is rewritten by hand rather than substituted. A parenthetical becomes brackets, an appositive becomes a colon, and a clause that was doing a full sentence's work becomes one. Also fixes a sentence in the RetroArch gate section whose subject drifted: the blind search watched PyBoy's work RAM to find the reference's channel addresses, and that fact was nested inside a clause about replaying through gambatte.
`gap()` sends one run-loop iteration between frame advances. That is a barrier, not a courtesy: `FRAMEADVANCE` returns as soon as RetroArch accepts it, so without an intervening iteration the next advance can be issued while the previous frame is still settling. `command` returns None when it gives up, and `gap` discarded that return. A barrier that did not happen looked exactly like one that did. This is a diagnostic as much as a fix. The same-process replay gate has failed four times today, and its divergence has a specific shape: two replays of one boot state and one input log agree byte for byte to emuFrame 811, then differ on channel values at IDENTICAL frame numbers. Identical frame numbers rule out a miscounted advance and point at a run loop that had not settled. It is NOT proven that a missed barrier causes it. `COMMAND_TIMEOUT` is 5 seconds with a resend loop, so a timeout there means RetroArch is wedged rather than busy, which should be rare. The fix decides the question: a gate that now fails with "the run loop was not synchronised" was this, and one that still diverges silently is something else. `save_state` and `load_state` also discard a command return, and are left alone deliberately. Both verify the outcome downstream, by waiting for the state file and by reading the log, which is a stronger check than the return.
The `authMode` column read empty on every authored cell. The row's `authMode` describes whoever played, and an authored cell is played by a program with no credentials at all, so that field was correctly null while the build behind it was very much billed to something. The authoring phase already read the value and then dropped it. It now travels in the `build` column, beside the tokens and minutes that were spent alongside it, and the runner prints the build cost on the cell line: minutes, tokens, and either a dollar figure or `unbilled(oauth)`. Measured on the six-cell study this fixes: sonnet spent 5.14M tokens building a player that lost the game after 317 decisions, while opus spent 1.19M on one that survived all 2000. Build cost and play score answer different questions and a single number cannot carry both.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The runtime was pinned at 0.142.3 against a current 0.175.0, and
@tangle-network/agent-evalwas not installed at all, so the optimizer this study wants to compose with could not be typechecked against.0.175.0 carries what a game needs.
VerifyResultgainedkeepGoingandscore. That is the disposition a score requires and a test suite does not: a green suite has nothing left to win, while a game always has a higher number above it.hillclimb.tsalready returns that exact shape.examples/agent-eval-fit.mtsis the proof, and it exists to be compiled. A playproof cell types as an agent-evalScenario— whereseedGroupis what makes two profiles on one game a paired comparison rather than two unrelated samples. Playing a candidate types as aVerifier. A change on either side that breaks the join now fails the typecheck instead of failing a study three hours in.Also fixes a flaky test the bump exposed, which was flaky before it. It spawned a child that exits, slept 120 ms, then asserted the driver had already been told the child was gone — which asserts how busy the machine is. It now waits for the condition, and passed 10 of 10 consecutive runs.
Local gate: 20 of 20 test files pass, boundary and attestation-blindness pass, 0 repo typecheck errors.