fix(headless): teach the trajectory exporter the two envelope fields it never learned by Astro-Han · Pull Request #2314 · apache/maka

Astro-Han · 2026-08-06T05:52:19Z

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
{ ..., "role": "model", "author": "agent",
"origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

	before	after
full trajectory	0	78
`runtime_event_schema_invalid`	89	0
`image_artifact_metadata_missing`	0	11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.

Astro-Han · 2026-08-06T05:52:19Z

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
{ ..., "role": "model", "author": "agent",
"origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

	before	after
full trajectory	0	78
`runtime_event_schema_invalid`	89	0
`image_artifact_metadata_missing`	0	11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.

Astro-Han · 2026-08-06T05:52:19Z

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
{ ..., "role": "model", "author": "agent",
"origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

	before	after
full trajectory	0	78
`runtime_event_schema_invalid`	89	0
`image_artifact_metadata_missing`	0	11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.

Astro-Han · 2026-08-06T05:52:19Z

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
{ ..., "role": "model", "author": "agent",
"origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

	before	after
full trajectory	0	78
`runtime_event_schema_invalid`	89	0
`image_artifact_metadata_missing`	0	11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.

Astro-Han · 2026-08-06T05:52:19Z

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
{ ..., "role": "model", "author": "agent",
"origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

	before	after
full trajectory	0	78
`runtime_event_schema_invalid`	89	0
`image_artifact_metadata_missing`	0	11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.

Astro-Han · 2026-08-06T05:52:19Z

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
{ ..., "role": "model", "author": "agent",
"origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

	before	after
full trajectory	0	78
`runtime_event_schema_invalid`	89	0
`image_artifact_metadata_missing`	0	11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.

Astro-Han · 2026-08-06T05:52:19Z

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
{ ..., "role": "model", "author": "agent",
"origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

	before	after
full trajectory	0	78
`runtime_event_schema_invalid`	89	0
`image_artifact_metadata_missing`	0	11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.

Astro-Han · 2026-08-06T05:52:19Z

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
{ ..., "role": "model", "author": "agent",
"origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

	before	after
full trajectory	0	78
`runtime_event_schema_invalid`	89	0
`image_artifact_metadata_missing`	0	11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.

Astro-Han added 2 commits August 6, 2026 14:25

test(core): count only accepted corpus cases as envelope-key coverage
41fd5c2

Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2Compare August 6, 2026 06:26

Astro-Han marked this pull request as ready for review August 6, 2026 06:36

Astro-Han merged commit 1e1c552 into mainAug 6, 2026
11 checks passed

This was referenced Aug 6, 2026
test(core): pin the shared validation corpus to every envelope value domain #2315
Merged
feat(headless): scan a finished run's trajectories for retrieval #2319
Merged

Astro-Han added 2 commits August 6, 2026 14:25

test(core): count only accepted corpus cases as envelope-key coverage
41fd5c2

Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2Compare August 6, 2026 06:26

Astro-Han marked this pull request as ready for review August 6, 2026 06:36

Astro-Han merged commit 1e1c552 into mainAug 6, 2026
11 checks passed

This was referenced Aug 6, 2026
test(core): pin the shared validation corpus to every envelope value domain #2315
Merged
feat(headless): scan a finished run's trajectories for retrieval #2319
Merged

Astro-Han added 2 commits August 6, 2026 14:25

test(core): count only accepted corpus cases as envelope-key coverage
41fd5c2

Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2Compare August 6, 2026 06:26

Astro-Han marked this pull request as ready for review August 6, 2026 06:36

Astro-Han merged commit 1e1c552 into mainAug 6, 2026
11 checks passed

This was referenced Aug 6, 2026
test(core): pin the shared validation corpus to every envelope value domain #2315
Merged
feat(headless): scan a finished run's trajectories for retrieval #2319
Merged

Astro-Han added 2 commits August 6, 2026 14:25

test(core): count only accepted corpus cases as envelope-key coverage
41fd5c2

Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2Compare August 6, 2026 06:26

Astro-Han marked this pull request as ready for review August 6, 2026 06:36

Astro-Han merged commit 1e1c552 into mainAug 6, 2026
11 checks passed

This was referenced Aug 6, 2026
test(core): pin the shared validation corpus to every envelope value domain #2315
Merged
feat(headless): scan a finished run's trajectories for retrieval #2319
Merged

Astro-Han added 2 commits August 6, 2026 14:25

test(core): count only accepted corpus cases as envelope-key coverage
41fd5c2

Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2Compare August 6, 2026 06:26

Astro-Han marked this pull request as ready for review August 6, 2026 06:36

Astro-Han merged commit 1e1c552 into mainAug 6, 2026
11 checks passed

This was referenced Aug 6, 2026
test(core): pin the shared validation corpus to every envelope value domain #2315
Merged
feat(headless): scan a finished run's trajectories for retrieval #2319
Merged

Astro-Han added 2 commits August 6, 2026 14:25

test(core): count only accepted corpus cases as envelope-key coverage
41fd5c2

Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2Compare August 6, 2026 06:26

Astro-Han marked this pull request as ready for review August 6, 2026 06:36

Astro-Han merged commit 1e1c552 into mainAug 6, 2026
11 checks passed

This was referenced Aug 6, 2026
test(core): pin the shared validation corpus to every envelope value domain #2315
Merged
feat(headless): scan a finished run's trajectories for retrieval #2319
Merged

Astro-Han added 2 commits August 6, 2026 14:25

test(core): count only accepted corpus cases as envelope-key coverage
41fd5c2

Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2Compare August 6, 2026 06:26

Astro-Han marked this pull request as ready for review August 6, 2026 06:36

Astro-Han merged commit 1e1c552 into mainAug 6, 2026
11 checks passed

This was referenced Aug 6, 2026
test(core): pin the shared validation corpus to every envelope value domain #2315
Merged
feat(headless): scan a finished run's trajectories for retrieval #2319
Merged

Astro-Han added 2 commits August 6, 2026 14:25

test(core): count only accepted corpus cases as envelope-key coverage
41fd5c2

Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2Compare August 6, 2026 06:26

Astro-Han marked this pull request as ready for review August 6, 2026 06:36

Astro-Han merged commit 1e1c552 into mainAug 6, 2026
11 checks passed

This was referenced Aug 6, 2026
test(core): pin the shared validation corpus to every envelope value domain #2315
Merged
feat(headless): scan a finished run's trajectories for retrieval #2319
Merged

Uh oh!

Conversation

Astro-Han commented Aug 6, 2026

What

Root cause

Why the existing guard missed it

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Astro-Han commented Aug 6, 2026

What

Root cause

Why the existing guard missed it

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Astro-Han commented Aug 6, 2026

What

Root cause

Why the existing guard missed it

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Astro-Han commented Aug 6, 2026

What

Root cause

Why the existing guard missed it

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Astro-Han commented Aug 6, 2026

What

Root cause

Why the existing guard missed it

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels