Skip to content

feat(matrix): keep everything a run measured, not only its headline - #45

Merged
drewstone merged 1 commit into
mainfrom
feat/telemetry
Aug 29, 2026
Merged

feat(matrix): keep everything a run measured, not only its headline#45
drewstone merged 1 commit into
mainfrom
feat/telemetry

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

A row answered "how did it do". Every autopsy in this work has needed "what happened" — and the data was already collected, then discarded at the row boundary. latencyMs was a full per-decision array. The evidence watcher held every channel at every decision. Driver health was a structured record being flattened into one sentence of prose.

The raw trace

Every decision, in order: the turn, the word played, that decision's latency, and every evidence channel as it stood afterwards.

turn 1 input=NOOP latency=82ms score=0 lives=5 ball=(0,0) paddle=146
turn 2 input=FIRE latency=0ms score=0 lives=5 ball=(124,116) paddle=146

On ALE Breakout that includes the ball and paddle RAM addresses, so you can watch the ball spawn at the exact decision that pressed FIRE. Written beside the study as JSONL, one decision per line — a 20,000-decision episode inside the row would make the study file unopenable.

Distributions, not means

spreadOf reports n, min, p50, p90, p99, max, mean and a twelve-bucket histogram.

Measured on one cell: 125 decisions at ~0 ms and one at 329 ms, for a mean of 3.20 that describes neither. A mean has hidden every interesting result in this work, including a profile that looked consistent to 0.2% at a censored horizon and varied 5.3× uncensored.

Also on the row now

The scored channel as a series; every channel's last value; input total against distinct inputs; per-milestone verification; the transport's health as data rather than prose; budget exhaustion; and the build phase's input tokens, output tokens and turns kept apart. That ratio is diagnostic — 1.8M in against 72 out is an agent reading, not one writing.

Artifacts survive

The authoring sandbox is a temp directory that gets reclaimed, so the program an agent wrote and the transcript of it writing were lost when the run ended. Both are copied beside the study now.

It found a defect immediately

The bundled masher emitted a hardcoded lowercase word. On Breakout, whose vocabulary is NOOP FIRE RIGHT LEFT, every input was illegal and the first decision spent ten seconds reaching a driver timeout. It reads its vocabulary from the request now, as the cycler already did — I had fixed one of the two controls and the trace caught the other.

Local: 20 of 20 test files, boundary check, 0 typecheck errors.

A row answered "how did it do". Every autopsy in this work has needed "what
happened", and the data to answer it was already collected and then discarded
at the row boundary: `latencyMs` was a full per-decision array, the evidence
watcher held every channel at every decision, and driver health was a
structured record being flattened into one sentence of prose.
THE RAW TRACE. Every decision, in order, carrying the turn, the word played,
that decision's latency, and every evidence channel as it stood afterwards. On
ALE Breakout that is the ball and paddle RAM addresses, so a reader can watch
the ball spawn at the exact decision that pressed FIRE. It is written beside
the study as JSONL, one decision per line, because a 20,000-decision episode
inside the row would make the study file unopenable.
DISTRIBUTIONS, NOT MEANS. `spreadOf` reports n, min, p50, p90, p99, max, mean
and a twelve-bucket histogram. Measured on one cell: 125 decisions at about
0 ms and one at 329 ms, a mean of 3.20 that describes neither. A mean has
hidden every interesting result here, including a profile that looked
consistent to 0.2% at a censored horizon and varied 5.3x uncensored.
WHAT ELSE THE ROW NOW CARRIES. The scored channel as a series, every channel's
last value, input totals against distinct inputs, per-milestone verification,
the transport's health as data, budget exhaustion, and the build phase's input
and output tokens and turns apart. That ratio is diagnostic: 1.8M in against
72 out is an agent reading, not one writing.
ARTIFACTS SURVIVE. The authoring sandbox is a temp directory that is
reclaimed, so the program an agent wrote and the transcript of it writing were
lost when the run ended. Both are copied beside the study now.
The trace found a defect immediately. The bundled masher emitted a hardcoded
lowercase word, so on Breakout every input was illegal and the first decision
spent ten seconds reaching a driver timeout. It reads its vocabulary from the
request now, as the cycler already did.
@drewstone
drewstone merged commit 3c7ebf9 into mainAug 29, 2026
6 checks passed
@drewstone
drewstone deleted the feat/telemetry branch August 29, 2026 09:31
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@drewstone