fix(headless): measure the wire the Maka runtime actually dials

[Astro-Han]

Summary

Two independent measurement bugs in the harness, both surfaced by the #2245 two-arm rerun.

1. The proxy measured a wire the Maka arm was not dialing. The provider proxy inferred its usage protocol from runtimeAdapter.kind, so every openai-compatible provider was parsed as Chat SSE. That inference went silently wrong when #2152 moved deepseek-v4-flash to the Responses API: the proxy never saw data: [DONE], recorded every request as interrupted with zero usage and no reasoning tokens, and incompleteTerminalProviderRequest then threw the whole graded cell away as an infra failure — cells that had in fact passed with reward 1.0.

The Maka arm runs the Maka runtime, so resolveModelRuntime is the authority on which wire it dials; ask it instead of guessing. Competitors run their own CLI and keep the adapter-kind path, which is what describes them. The A/B manifest's per-arm transport comes from the same call, so a run now records the protocol it measured rather than the one it assumed — the frozen three-way composition test asserted openai-chat for the Maka arm, describing a run that never happened.

packages/runtime gains a ./model-runtime export so the harness can reach the resolver.

2. The infra-retry allowlist could not name a dotted task.executeHarnessArmCohort spelled candidate round ids as ab-${arm}-r0-${task.id} by hand, while buildAbRoundId — the helper that actually writes them — normalizes . to -. A retry naming install-windows-3.11 was therefore rejected as unknown, and because the check throws rather than skips, that one task took the whole batch down with it: a rerun of three infra-failed cells exited in twelve seconds having run none of them, twice. Building the candidates with the same helper removes the second spelling.

Refs #2245.

Verification

npm run -w @maka/headless test — 1424 pass / 0 fail. npm run -w @maka/runtime test — 3270 pass / 0 fail. npm run lint and npm run format clean.

The dotted-retry regression test was verified red before the fix (adjudicated infra retry names unknown round ab-maka-r0-install-windows-3-11) and green after.

Reproduced (1) against the live 5-task Terminal-Bench 2.1 pilot on deepseek-v4-flash (the same seed order as the 2026-08-04 four-arm run, so count-dataset-tokens is the same cell in all three columns):

	2026-08-04 (Chat wire)	before this fix	after
path / parsed protocol	/chat/completions · chat-sse	/responses · chat-sse	/responses · responses-sse
requests	24, completed 24	20, interrupted 20	24, completed 24
reasoning tokens	3358	null (0 measured)	5154 (24/24 measured)
cell verdict	pass	infra_failed	completed, pass

A second pilot cell (regex-log) is likewise 15/15 completed with 17058 reasoning tokens measured.

The full 89-task two-arm run then completed on this branch: 3479 Maka requests over 89 cells with 9 non-completed, versus every request non-completed before the fix.

Root cause

Not #2241/#2249. fc581d855 (#2152, provider-native web search) added deepseek-v4-flash → openai-responses to openAiAdapterApiProtocol. The 2026-08-04 benchmark ran on 9dae4bee, which predates it and therefore dialled Chat Completions; the wire changed under a harness that kept guessing the old one.

Review round 2

Review found the fix's own API could re-admit the bug it closes. providerProxyUsageProtocol took the model id as an optional, prefix-sensitive argument, and the two call sites applied the discipline while the manifest call did not. resolveModelRuntime does not recognize the catalog spelling deepseek/deepseek-v4-flash, so a caller forwarding it raw silently got the Chat guess back — the exact wrong number this PR exists to remove — and a caller passing nothing at all got it too, with TypeScript unable to say so.

The function now normalizes the id itself and refuses to guess for the maka arm. That deletes the caller-side discipline at both runners rather than testing it: there is no longer a spelling a call site can get wrong. Two assertions pin it, including the assert.throws for a missing id.

Unknown retry round ids are also now reported together, so an operator recovering a sweep with a batch of ids fixes every typo in one pass.

[Astro-Han]

Summary

Two independent measurement bugs in the harness, both surfaced by the #2245 two-arm rerun.

1. The proxy measured a wire the Maka arm was not dialing. The provider proxy inferred its usage protocol from runtimeAdapter.kind, so every openai-compatible provider was parsed as Chat SSE. That inference went silently wrong when #2152 moved deepseek-v4-flash to the Responses API: the proxy never saw data: [DONE], recorded every request as interrupted with zero usage and no reasoning tokens, and incompleteTerminalProviderRequest then threw the whole graded cell away as an infra failure — cells that had in fact passed with reward 1.0.

The Maka arm runs the Maka runtime, so resolveModelRuntime is the authority on which wire it dials; ask it instead of guessing. Competitors run their own CLI and keep the adapter-kind path, which is what describes them. The A/B manifest's per-arm transport comes from the same call, so a run now records the protocol it measured rather than the one it assumed — the frozen three-way composition test asserted openai-chat for the Maka arm, describing a run that never happened.

packages/runtime gains a ./model-runtime export so the harness can reach the resolver.

2. The infra-retry allowlist could not name a dotted task.executeHarnessArmCohort spelled candidate round ids as ab-${arm}-r0-${task.id} by hand, while buildAbRoundId — the helper that actually writes them — normalizes . to -. A retry naming install-windows-3.11 was therefore rejected as unknown, and because the check throws rather than skips, that one task took the whole batch down with it: a rerun of three infra-failed cells exited in twelve seconds having run none of them, twice. Building the candidates with the same helper removes the second spelling.

Refs #2245.

Verification

npm run -w @maka/headless test — 1424 pass / 0 fail. npm run -w @maka/runtime test — 3270 pass / 0 fail. npm run lint and npm run format clean.

The dotted-retry regression test was verified red before the fix (adjudicated infra retry names unknown round ab-maka-r0-install-windows-3-11) and green after.

Reproduced (1) against the live 5-task Terminal-Bench 2.1 pilot on deepseek-v4-flash (the same seed order as the 2026-08-04 four-arm run, so count-dataset-tokens is the same cell in all three columns):

	2026-08-04 (Chat wire)	before this fix	after
path / parsed protocol	/chat/completions · chat-sse	/responses · chat-sse	/responses · responses-sse
requests	24, completed 24	20, interrupted 20	24, completed 24
reasoning tokens	3358	null (0 measured)	5154 (24/24 measured)
cell verdict	pass	infra_failed	completed, pass

A second pilot cell (regex-log) is likewise 15/15 completed with 17058 reasoning tokens measured.

The full 89-task two-arm run then completed on this branch: 3479 Maka requests over 89 cells with 9 non-completed, versus every request non-completed before the fix.

Root cause

Not #2241/#2249. fc581d855 (#2152, provider-native web search) added deepseek-v4-flash → openai-responses to openAiAdapterApiProtocol. The 2026-08-04 benchmark ran on 9dae4bee, which predates it and therefore dialled Chat Completions; the wire changed under a harness that kept guessing the old one.

Review round 2

Review found the fix's own API could re-admit the bug it closes. providerProxyUsageProtocol took the model id as an optional, prefix-sensitive argument, and the two call sites applied the discipline while the manifest call did not. resolveModelRuntime does not recognize the catalog spelling deepseek/deepseek-v4-flash, so a caller forwarding it raw silently got the Chat guess back — the exact wrong number this PR exists to remove — and a caller passing nothing at all got it too, with TypeScript unable to say so.

The function now normalizes the id itself and refuses to guess for the maka arm. That deletes the caller-side discipline at both runners rather than testing it: there is no longer a spelling a call site can get wrong. Two assertions pin it, including the assert.throws for a missing id.

Unknown retry round ids are also now reported together, so an operator recovering a sweep with a batch of ids fixes every typo in one pass.

[Astro-Han]

Summary

Two independent measurement bugs in the harness, both surfaced by the #2245 two-arm rerun.

1. The proxy measured a wire the Maka arm was not dialing. The provider proxy inferred its usage protocol from runtimeAdapter.kind, so every openai-compatible provider was parsed as Chat SSE. That inference went silently wrong when #2152 moved deepseek-v4-flash to the Responses API: the proxy never saw data: [DONE], recorded every request as interrupted with zero usage and no reasoning tokens, and incompleteTerminalProviderRequest then threw the whole graded cell away as an infra failure — cells that had in fact passed with reward 1.0.

The Maka arm runs the Maka runtime, so resolveModelRuntime is the authority on which wire it dials; ask it instead of guessing. Competitors run their own CLI and keep the adapter-kind path, which is what describes them. The A/B manifest's per-arm transport comes from the same call, so a run now records the protocol it measured rather than the one it assumed — the frozen three-way composition test asserted openai-chat for the Maka arm, describing a run that never happened.

packages/runtime gains a ./model-runtime export so the harness can reach the resolver.

2. The infra-retry allowlist could not name a dotted task.executeHarnessArmCohort spelled candidate round ids as ab-${arm}-r0-${task.id} by hand, while buildAbRoundId — the helper that actually writes them — normalizes . to -. A retry naming install-windows-3.11 was therefore rejected as unknown, and because the check throws rather than skips, that one task took the whole batch down with it: a rerun of three infra-failed cells exited in twelve seconds having run none of them, twice. Building the candidates with the same helper removes the second spelling.

Refs #2245.

Verification

npm run -w @maka/headless test — 1424 pass / 0 fail. npm run -w @maka/runtime test — 3270 pass / 0 fail. npm run lint and npm run format clean.

The dotted-retry regression test was verified red before the fix (adjudicated infra retry names unknown round ab-maka-r0-install-windows-3-11) and green after.

Reproduced (1) against the live 5-task Terminal-Bench 2.1 pilot on deepseek-v4-flash (the same seed order as the 2026-08-04 four-arm run, so count-dataset-tokens is the same cell in all three columns):

	2026-08-04 (Chat wire)	before this fix	after
path / parsed protocol	/chat/completions · chat-sse	/responses · chat-sse	/responses · responses-sse
requests	24, completed 24	20, interrupted 20	24, completed 24
reasoning tokens	3358	null (0 measured)	5154 (24/24 measured)
cell verdict	pass	infra_failed	completed, pass

A second pilot cell (regex-log) is likewise 15/15 completed with 17058 reasoning tokens measured.

The full 89-task two-arm run then completed on this branch: 3479 Maka requests over 89 cells with 9 non-completed, versus every request non-completed before the fix.

Root cause

Not #2241/#2249. fc581d855 (#2152, provider-native web search) added deepseek-v4-flash → openai-responses to openAiAdapterApiProtocol. The 2026-08-04 benchmark ran on 9dae4bee, which predates it and therefore dialled Chat Completions; the wire changed under a harness that kept guessing the old one.

Review round 2

Review found the fix's own API could re-admit the bug it closes. providerProxyUsageProtocol took the model id as an optional, prefix-sensitive argument, and the two call sites applied the discipline while the manifest call did not. resolveModelRuntime does not recognize the catalog spelling deepseek/deepseek-v4-flash, so a caller forwarding it raw silently got the Chat guess back — the exact wrong number this PR exists to remove — and a caller passing nothing at all got it too, with TypeScript unable to say so.

The function now normalizes the id itself and refuses to guess for the maka arm. That deletes the caller-side discipline at both runners rather than testing it: there is no longer a spelling a call site can get wrong. Two assertions pin it, including the assert.throws for a missing id.

Unknown retry round ids are also now reported together, so an operator recovering a sweep with a batch of ids fixes every typo in one pass.

[Astro-Han]

Summary

Two independent measurement bugs in the harness, both surfaced by the #2245 two-arm rerun.

1. The proxy measured a wire the Maka arm was not dialing. The provider proxy inferred its usage protocol from runtimeAdapter.kind, so every openai-compatible provider was parsed as Chat SSE. That inference went silently wrong when #2152 moved deepseek-v4-flash to the Responses API: the proxy never saw data: [DONE], recorded every request as interrupted with zero usage and no reasoning tokens, and incompleteTerminalProviderRequest then threw the whole graded cell away as an infra failure — cells that had in fact passed with reward 1.0.

The Maka arm runs the Maka runtime, so resolveModelRuntime is the authority on which wire it dials; ask it instead of guessing. Competitors run their own CLI and keep the adapter-kind path, which is what describes them. The A/B manifest's per-arm transport comes from the same call, so a run now records the protocol it measured rather than the one it assumed — the frozen three-way composition test asserted openai-chat for the Maka arm, describing a run that never happened.

packages/runtime gains a ./model-runtime export so the harness can reach the resolver.

2. The infra-retry allowlist could not name a dotted task.executeHarnessArmCohort spelled candidate round ids as ab-${arm}-r0-${task.id} by hand, while buildAbRoundId — the helper that actually writes them — normalizes . to -. A retry naming install-windows-3.11 was therefore rejected as unknown, and because the check throws rather than skips, that one task took the whole batch down with it: a rerun of three infra-failed cells exited in twelve seconds having run none of them, twice. Building the candidates with the same helper removes the second spelling.

Refs #2245.

Verification

npm run -w @maka/headless test — 1424 pass / 0 fail. npm run -w @maka/runtime test — 3270 pass / 0 fail. npm run lint and npm run format clean.

The dotted-retry regression test was verified red before the fix (adjudicated infra retry names unknown round ab-maka-r0-install-windows-3-11) and green after.

Reproduced (1) against the live 5-task Terminal-Bench 2.1 pilot on deepseek-v4-flash (the same seed order as the 2026-08-04 four-arm run, so count-dataset-tokens is the same cell in all three columns):

	2026-08-04 (Chat wire)	before this fix	after
path / parsed protocol	/chat/completions · chat-sse	/responses · chat-sse	/responses · responses-sse
requests	24, completed 24	20, interrupted 20	24, completed 24
reasoning tokens	3358	null (0 measured)	5154 (24/24 measured)
cell verdict	pass	infra_failed	completed, pass

A second pilot cell (regex-log) is likewise 15/15 completed with 17058 reasoning tokens measured.

The full 89-task two-arm run then completed on this branch: 3479 Maka requests over 89 cells with 9 non-completed, versus every request non-completed before the fix.

Root cause

Not #2241/#2249. fc581d855 (#2152, provider-native web search) added deepseek-v4-flash → openai-responses to openAiAdapterApiProtocol. The 2026-08-04 benchmark ran on 9dae4bee, which predates it and therefore dialled Chat Completions; the wire changed under a harness that kept guessing the old one.

Review round 2

Review found the fix's own API could re-admit the bug it closes. providerProxyUsageProtocol took the model id as an optional, prefix-sensitive argument, and the two call sites applied the discipline while the manifest call did not. resolveModelRuntime does not recognize the catalog spelling deepseek/deepseek-v4-flash, so a caller forwarding it raw silently got the Chat guess back — the exact wrong number this PR exists to remove — and a caller passing nothing at all got it too, with TypeScript unable to say so.

The function now normalizes the id itself and refuses to guess for the maka arm. That deletes the caller-side discipline at both runners rather than testing it: there is no longer a spelling a call site can get wrong. Two assertions pin it, including the assert.throws for a missing id.

Unknown retry round ids are also now reported together, so an operator recovering a sweep with a batch of ids fixes every typo in one pass.

[Astro-Han]

Summary

Two independent measurement bugs in the harness, both surfaced by the #2245 two-arm rerun.

1. The proxy measured a wire the Maka arm was not dialing. The provider proxy inferred its usage protocol from runtimeAdapter.kind, so every openai-compatible provider was parsed as Chat SSE. That inference went silently wrong when #2152 moved deepseek-v4-flash to the Responses API: the proxy never saw data: [DONE], recorded every request as interrupted with zero usage and no reasoning tokens, and incompleteTerminalProviderRequest then threw the whole graded cell away as an infra failure — cells that had in fact passed with reward 1.0.

The Maka arm runs the Maka runtime, so resolveModelRuntime is the authority on which wire it dials; ask it instead of guessing. Competitors run their own CLI and keep the adapter-kind path, which is what describes them. The A/B manifest's per-arm transport comes from the same call, so a run now records the protocol it measured rather than the one it assumed — the frozen three-way composition test asserted openai-chat for the Maka arm, describing a run that never happened.

packages/runtime gains a ./model-runtime export so the harness can reach the resolver.

2. The infra-retry allowlist could not name a dotted task.executeHarnessArmCohort spelled candidate round ids as ab-${arm}-r0-${task.id} by hand, while buildAbRoundId — the helper that actually writes them — normalizes . to -. A retry naming install-windows-3.11 was therefore rejected as unknown, and because the check throws rather than skips, that one task took the whole batch down with it: a rerun of three infra-failed cells exited in twelve seconds having run none of them, twice. Building the candidates with the same helper removes the second spelling.

Refs #2245.

Verification

npm run -w @maka/headless test — 1424 pass / 0 fail. npm run -w @maka/runtime test — 3270 pass / 0 fail. npm run lint and npm run format clean.

The dotted-retry regression test was verified red before the fix (adjudicated infra retry names unknown round ab-maka-r0-install-windows-3-11) and green after.

Reproduced (1) against the live 5-task Terminal-Bench 2.1 pilot on deepseek-v4-flash (the same seed order as the 2026-08-04 four-arm run, so count-dataset-tokens is the same cell in all three columns):

	2026-08-04 (Chat wire)	before this fix	after
path / parsed protocol	/chat/completions · chat-sse	/responses · chat-sse	/responses · responses-sse
requests	24, completed 24	20, interrupted 20	24, completed 24
reasoning tokens	3358	null (0 measured)	5154 (24/24 measured)
cell verdict	pass	infra_failed	completed, pass

A second pilot cell (regex-log) is likewise 15/15 completed with 17058 reasoning tokens measured.

The full 89-task two-arm run then completed on this branch: 3479 Maka requests over 89 cells with 9 non-completed, versus every request non-completed before the fix.

Root cause

Not #2241/#2249. fc581d855 (#2152, provider-native web search) added deepseek-v4-flash → openai-responses to openAiAdapterApiProtocol. The 2026-08-04 benchmark ran on 9dae4bee, which predates it and therefore dialled Chat Completions; the wire changed under a harness that kept guessing the old one.

Review round 2

Review found the fix's own API could re-admit the bug it closes. providerProxyUsageProtocol took the model id as an optional, prefix-sensitive argument, and the two call sites applied the discipline while the manifest call did not. resolveModelRuntime does not recognize the catalog spelling deepseek/deepseek-v4-flash, so a caller forwarding it raw silently got the Chat guess back — the exact wrong number this PR exists to remove — and a caller passing nothing at all got it too, with TypeScript unable to say so.

The function now normalizes the id itself and refuses to guess for the maka arm. That deletes the caller-side discipline at both runners rather than testing it: there is no longer a spelling a call site can get wrong. Two assertions pin it, including the assert.throws for a missing id.

Unknown retry round ids are also now reported together, so an operator recovering a sweep with a batch of ids fixes every typo in one pass.

[Astro-Han]

Summary

Two independent measurement bugs in the harness, both surfaced by the #2245 two-arm rerun.

1. The proxy measured a wire the Maka arm was not dialing. The provider proxy inferred its usage protocol from runtimeAdapter.kind, so every openai-compatible provider was parsed as Chat SSE. That inference went silently wrong when #2152 moved deepseek-v4-flash to the Responses API: the proxy never saw data: [DONE], recorded every request as interrupted with zero usage and no reasoning tokens, and incompleteTerminalProviderRequest then threw the whole graded cell away as an infra failure — cells that had in fact passed with reward 1.0.

The Maka arm runs the Maka runtime, so resolveModelRuntime is the authority on which wire it dials; ask it instead of guessing. Competitors run their own CLI and keep the adapter-kind path, which is what describes them. The A/B manifest's per-arm transport comes from the same call, so a run now records the protocol it measured rather than the one it assumed — the frozen three-way composition test asserted openai-chat for the Maka arm, describing a run that never happened.

packages/runtime gains a ./model-runtime export so the harness can reach the resolver.

2. The infra-retry allowlist could not name a dotted task.executeHarnessArmCohort spelled candidate round ids as ab-${arm}-r0-${task.id} by hand, while buildAbRoundId — the helper that actually writes them — normalizes . to -. A retry naming install-windows-3.11 was therefore rejected as unknown, and because the check throws rather than skips, that one task took the whole batch down with it: a rerun of three infra-failed cells exited in twelve seconds having run none of them, twice. Building the candidates with the same helper removes the second spelling.

Refs #2245.

Verification

npm run -w @maka/headless test — 1424 pass / 0 fail. npm run -w @maka/runtime test — 3270 pass / 0 fail. npm run lint and npm run format clean.

The dotted-retry regression test was verified red before the fix (adjudicated infra retry names unknown round ab-maka-r0-install-windows-3-11) and green after.

Reproduced (1) against the live 5-task Terminal-Bench 2.1 pilot on deepseek-v4-flash (the same seed order as the 2026-08-04 four-arm run, so count-dataset-tokens is the same cell in all three columns):

	2026-08-04 (Chat wire)	before this fix	after
path / parsed protocol	/chat/completions · chat-sse	/responses · chat-sse	/responses · responses-sse
requests	24, completed 24	20, interrupted 20	24, completed 24
reasoning tokens	3358	null (0 measured)	5154 (24/24 measured)
cell verdict	pass	infra_failed	completed, pass

A second pilot cell (regex-log) is likewise 15/15 completed with 17058 reasoning tokens measured.

The full 89-task two-arm run then completed on this branch: 3479 Maka requests over 89 cells with 9 non-completed, versus every request non-completed before the fix.

Root cause

Not #2241/#2249. fc581d855 (#2152, provider-native web search) added deepseek-v4-flash → openai-responses to openAiAdapterApiProtocol. The 2026-08-04 benchmark ran on 9dae4bee, which predates it and therefore dialled Chat Completions; the wire changed under a harness that kept guessing the old one.

Review round 2

Review found the fix's own API could re-admit the bug it closes. providerProxyUsageProtocol took the model id as an optional, prefix-sensitive argument, and the two call sites applied the discipline while the manifest call did not. resolveModelRuntime does not recognize the catalog spelling deepseek/deepseek-v4-flash, so a caller forwarding it raw silently got the Chat guess back — the exact wrong number this PR exists to remove — and a caller passing nothing at all got it too, with TypeScript unable to say so.

The function now normalizes the id itself and refuses to guess for the maka arm. That deletes the caller-side discipline at both runners rather than testing it: there is no longer a spelling a call site can get wrong. Two assertions pin it, including the assert.throws for a missing id.

Unknown retry round ids are also now reported together, so an operator recovering a sweep with a batch of ids fixes every typo in one pass.

[Astro-Han]

Summary

Two independent measurement bugs in the harness, both surfaced by the #2245 two-arm rerun.

1. The proxy measured a wire the Maka arm was not dialing. The provider proxy inferred its usage protocol from runtimeAdapter.kind, so every openai-compatible provider was parsed as Chat SSE. That inference went silently wrong when #2152 moved deepseek-v4-flash to the Responses API: the proxy never saw data: [DONE], recorded every request as interrupted with zero usage and no reasoning tokens, and incompleteTerminalProviderRequest then threw the whole graded cell away as an infra failure — cells that had in fact passed with reward 1.0.

The Maka arm runs the Maka runtime, so resolveModelRuntime is the authority on which wire it dials; ask it instead of guessing. Competitors run their own CLI and keep the adapter-kind path, which is what describes them. The A/B manifest's per-arm transport comes from the same call, so a run now records the protocol it measured rather than the one it assumed — the frozen three-way composition test asserted openai-chat for the Maka arm, describing a run that never happened.

packages/runtime gains a ./model-runtime export so the harness can reach the resolver.

2. The infra-retry allowlist could not name a dotted task.executeHarnessArmCohort spelled candidate round ids as ab-${arm}-r0-${task.id} by hand, while buildAbRoundId — the helper that actually writes them — normalizes . to -. A retry naming install-windows-3.11 was therefore rejected as unknown, and because the check throws rather than skips, that one task took the whole batch down with it: a rerun of three infra-failed cells exited in twelve seconds having run none of them, twice. Building the candidates with the same helper removes the second spelling.

Refs #2245.

Verification

npm run -w @maka/headless test — 1424 pass / 0 fail. npm run -w @maka/runtime test — 3270 pass / 0 fail. npm run lint and npm run format clean.

The dotted-retry regression test was verified red before the fix (adjudicated infra retry names unknown round ab-maka-r0-install-windows-3-11) and green after.

Reproduced (1) against the live 5-task Terminal-Bench 2.1 pilot on deepseek-v4-flash (the same seed order as the 2026-08-04 four-arm run, so count-dataset-tokens is the same cell in all three columns):

	2026-08-04 (Chat wire)	before this fix	after
path / parsed protocol	/chat/completions · chat-sse	/responses · chat-sse	/responses · responses-sse
requests	24, completed 24	20, interrupted 20	24, completed 24
reasoning tokens	3358	null (0 measured)	5154 (24/24 measured)
cell verdict	pass	infra_failed	completed, pass

A second pilot cell (regex-log) is likewise 15/15 completed with 17058 reasoning tokens measured.

The full 89-task two-arm run then completed on this branch: 3479 Maka requests over 89 cells with 9 non-completed, versus every request non-completed before the fix.

Root cause

Not #2241/#2249. fc581d855 (#2152, provider-native web search) added deepseek-v4-flash → openai-responses to openAiAdapterApiProtocol. The 2026-08-04 benchmark ran on 9dae4bee, which predates it and therefore dialled Chat Completions; the wire changed under a harness that kept guessing the old one.

Review round 2

Review found the fix's own API could re-admit the bug it closes. providerProxyUsageProtocol took the model id as an optional, prefix-sensitive argument, and the two call sites applied the discipline while the manifest call did not. resolveModelRuntime does not recognize the catalog spelling deepseek/deepseek-v4-flash, so a caller forwarding it raw silently got the Chat guess back — the exact wrong number this PR exists to remove — and a caller passing nothing at all got it too, with TypeScript unable to say so.

The function now normalizes the id itself and refuses to guess for the maka arm. That deletes the caller-side discipline at both runners rather than testing it: there is no longer a spelling a call site can get wrong. Two assertions pin it, including the assert.throws for a missing id.

Unknown retry round ids are also now reported together, so an operator recovering a sweep with a batch of ids fixes every typo in one pass.

[Astro-Han]

Summary

Two independent measurement bugs in the harness, both surfaced by the #2245 two-arm rerun.

1. The proxy measured a wire the Maka arm was not dialing. The provider proxy inferred its usage protocol from runtimeAdapter.kind, so every openai-compatible provider was parsed as Chat SSE. That inference went silently wrong when #2152 moved deepseek-v4-flash to the Responses API: the proxy never saw data: [DONE], recorded every request as interrupted with zero usage and no reasoning tokens, and incompleteTerminalProviderRequest then threw the whole graded cell away as an infra failure — cells that had in fact passed with reward 1.0.

The Maka arm runs the Maka runtime, so resolveModelRuntime is the authority on which wire it dials; ask it instead of guessing. Competitors run their own CLI and keep the adapter-kind path, which is what describes them. The A/B manifest's per-arm transport comes from the same call, so a run now records the protocol it measured rather than the one it assumed — the frozen three-way composition test asserted openai-chat for the Maka arm, describing a run that never happened.

packages/runtime gains a ./model-runtime export so the harness can reach the resolver.

2. The infra-retry allowlist could not name a dotted task.executeHarnessArmCohort spelled candidate round ids as ab-${arm}-r0-${task.id} by hand, while buildAbRoundId — the helper that actually writes them — normalizes . to -. A retry naming install-windows-3.11 was therefore rejected as unknown, and because the check throws rather than skips, that one task took the whole batch down with it: a rerun of three infra-failed cells exited in twelve seconds having run none of them, twice. Building the candidates with the same helper removes the second spelling.

Refs #2245.

Verification

npm run -w @maka/headless test — 1424 pass / 0 fail. npm run -w @maka/runtime test — 3270 pass / 0 fail. npm run lint and npm run format clean.

The dotted-retry regression test was verified red before the fix (adjudicated infra retry names unknown round ab-maka-r0-install-windows-3-11) and green after.

Reproduced (1) against the live 5-task Terminal-Bench 2.1 pilot on deepseek-v4-flash (the same seed order as the 2026-08-04 four-arm run, so count-dataset-tokens is the same cell in all three columns):

	2026-08-04 (Chat wire)	before this fix	after
path / parsed protocol	/chat/completions · chat-sse	/responses · chat-sse	/responses · responses-sse
requests	24, completed 24	20, interrupted 20	24, completed 24
reasoning tokens	3358	null (0 measured)	5154 (24/24 measured)
cell verdict	pass	infra_failed	completed, pass

A second pilot cell (regex-log) is likewise 15/15 completed with 17058 reasoning tokens measured.

The full 89-task two-arm run then completed on this branch: 3479 Maka requests over 89 cells with 9 non-completed, versus every request non-completed before the fix.

Root cause

Not #2241/#2249. fc581d855 (#2152, provider-native web search) added deepseek-v4-flash → openai-responses to openAiAdapterApiProtocol. The 2026-08-04 benchmark ran on 9dae4bee, which predates it and therefore dialled Chat Completions; the wire changed under a harness that kept guessing the old one.

Review round 2

Review found the fix's own API could re-admit the bug it closes. providerProxyUsageProtocol took the model id as an optional, prefix-sensitive argument, and the two call sites applied the discipline while the manifest call did not. resolveModelRuntime does not recognize the catalog spelling deepseek/deepseek-v4-flash, so a caller forwarding it raw silently got the Chat guess back — the exact wrong number this PR exists to remove — and a caller passing nothing at all got it too, with TypeScript unable to say so.

The function now normalizes the id itself and refuses to guess for the maka arm. That deletes the caller-side discipline at both runners rather than testing it: there is no longer a spelling a call site can get wrong. Two assertions pin it, including the assert.throws for a missing id.

Unknown retry round ids are also now reported together, so an operator recovering a sweep with a batch of ids fixes every typo in one pass.