Uh oh!
There was an error while loading. Please reload this page.
feat(matrix): keep everything a run measured, not only its headline - #45
Merged
Conversation
A row answered "how did it do". Every autopsy in this work has needed "what happened", and the data to answer it was already collected and then discarded at the row boundary: `latencyMs` was a full per-decision array, the evidence watcher held every channel at every decision, and driver health was a structured record being flattened into one sentence of prose. THE RAW TRACE. Every decision, in order, carrying the turn, the word played, that decision's latency, and every evidence channel as it stood afterwards. On ALE Breakout that is the ball and paddle RAM addresses, so a reader can watch the ball spawn at the exact decision that pressed FIRE. It is written beside the study as JSONL, one decision per line, because a 20,000-decision episode inside the row would make the study file unopenable. DISTRIBUTIONS, NOT MEANS. `spreadOf` reports n, min, p50, p90, p99, max, mean and a twelve-bucket histogram. Measured on one cell: 125 decisions at about 0 ms and one at 329 ms, a mean of 3.20 that describes neither. A mean has hidden every interesting result here, including a profile that looked consistent to 0.2% at a censored horizon and varied 5.3x uncensored. WHAT ELSE THE ROW NOW CARRIES. The scored channel as a series, every channel's last value, input totals against distinct inputs, per-milestone verification, the transport's health as data, budget exhaustion, and the build phase's input and output tokens and turns apart. That ratio is diagnostic: 1.8M in against 72 out is an agent reading, not one writing. ARTIFACTS SURVIVE. The authoring sandbox is a temp directory that is reclaimed, so the program an agent wrote and the transcript of it writing were lost when the run ended. Both are copied beside the study now. The trace found a defect immediately. The bundled masher emitted a hardcoded lowercase word, so on Breakout every input was illegal and the first decision spent ten seconds reaching a driver timeout. It reads its vocabulary from the request now, as the cycler already did.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A row answered "how did it do". Every autopsy in this work has needed "what happened" — and the data was already collected, then discarded at the row boundary.
latencyMswas a full per-decision array. The evidence watcher held every channel at every decision. Driver health was a structured record being flattened into one sentence of prose.The raw trace
Every decision, in order: the turn, the word played, that decision's latency, and every evidence channel as it stood afterwards.
On ALE Breakout that includes the ball and paddle RAM addresses, so you can watch the ball spawn at the exact decision that pressed
FIRE. Written beside the study as JSONL, one decision per line — a 20,000-decision episode inside the row would make the study file unopenable.Distributions, not means
spreadOfreports n, min, p50, p90, p99, max, mean and a twelve-bucket histogram.Measured on one cell: 125 decisions at ~0 ms and one at 329 ms, for a mean of 3.20 that describes neither. A mean has hidden every interesting result in this work, including a profile that looked consistent to 0.2% at a censored horizon and varied 5.3× uncensored.
Also on the row now
The scored channel as a series; every channel's last value; input total against distinct inputs; per-milestone verification; the transport's health as data rather than prose; budget exhaustion; and the build phase's input tokens, output tokens and turns kept apart. That ratio is diagnostic — 1.8M in against 72 out is an agent reading, not one writing.
Artifacts survive
The authoring sandbox is a temp directory that gets reclaimed, so the program an agent wrote and the transcript of it writing were lost when the run ended. Both are copied beside the study now.
It found a defect immediately
The bundled
masheremitted a hardcoded lowercase word. On Breakout, whose vocabulary isNOOP FIRE RIGHT LEFT, every input was illegal and the first decision spent ten seconds reaching a driver timeout. It reads its vocabulary from the request now, as thecycleralready did — I had fixed one of the two controls and the trace caught the other.Local: 20 of 20 test files, boundary check, 0 typecheck errors.