Uh oh!
There was an error while loading. Please reload this page.
feat(matrix): state the vocabulary, and let two games have two clocks - #44
Merged
Conversation
Three faults, all found by running the first two-game study this repo has completed. A POLICY WAS NEVER TOLD WHAT WORDS THE GAME TAKES. The streaming transport states them in `brief.json`, exactly so an agent does not have to guess. The persistent and per-decision transports sent frame, history and context and never said. Measured: a control that cycles 2048's four words scored 1948 there and 0 on ALE Breakout across three replicates, with `distinctInputs=1` and no lives lost, because Breakout takes NOOP, FIRE, RIGHT and LEFT and none of the four it emitted was legal. It read as a player that did nothing rather than one that was never told the rules. Both transports now carry `commands` in the request, and the bundled control reads them instead of assuming. With the words stated it launches the ball and loses five lives in 126 decisions. `assertJoinable` REFUSED THE STUDY IT EXISTS TO SERVE. It rejected two protocols across the whole pool, and two games legitimately have two clocks: a packaged reference pins the clock it was recorded at, so Breakout is derivable only at frameskip 4 while native-2048 runs at 1. A transfer statistic ranks profiles inside each game and correlates the orders, so a raw score never crosses a game boundary. The check is now scoped per game. Two clocks inside ONE game is still refused, because those rows are averaged and the dominant term disappears into the mean. The bundled controls lived under `runs/`, which is gitignored, so a definition naming them was broken for everyone but its author. They are in `examples/` now, with the two halves of the study that found all of this. Measured end to end: 12 cells, 0 blocked, and the first cross-game statistic this repository has produced. It reports `tau=0.000` and says why that is not a finding: both controls score 0 on Breakout, so there is no order to correlate, and one game pair is a single observation.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three faults, all found by running the first two-game study this repo has completed.
A policy was never told what words the game takes
The streaming transport states them in
brief.json, exactly so an agent does not have to guess. The persistent and per-decision transports sentframe,historyandcontextand never said.Measured: a control that cycles 2048's four words scored 1948 there and 0 on ALE Breakout across three replicates, with
distinctInputs=1and no lives lost — Breakout takesNOOP FIRE RIGHT LEFTand none of the four it emitted was legal. Every word was substituted, so it read as a player that did nothing rather than one that was never told the rules.Both transports now carry
commandsin the request, and the bundled control reads them instead of assuming. With the words stated it launches the ball and loses five lives in 126 decisions.assertJoinablerefused the study it exists to serveIt rejected "2 protocols" across the whole pool. Two games legitimately have two clocks: a packaged reference pins the clock it was recorded at, so Breakout is derivable only at
frameskip 4while native-2048 runs at1.A transfer statistic ranks profiles inside each game and correlates the orders, so a raw score never crosses a game boundary. The check is now scoped per game. Two clocks inside one game is still refused — those rows get averaged and the dominant term disappears into the mean.
The bundled controls were unreachable
They lived under
runs/, which is gitignored, so any definition naming them was broken for everyone but its author. They are inexamples/policies/now, alongside the two halves of the study that found all of this.Result
12 cells, 0 blocked, and the first cross-game statistic this repository has produced:
The machinery works end to end and the study is uninformative, which are different things. Both controls score 0 on Breakout, so there is no order to correlate; the runner says so rather than presenting the zero as a finding. Separating profiles there needs agents, not cheap controls — neither control can hit a ball.
Local: 20 of 20 test files, boundary check, 0 typecheck errors.