feat(benchmark): compare Kimi coding protocols

[Nyvo-io]

Summary

• keep this PR as the benchmark-harness layer requested in the maintainer split
• compare the existing Kimi Coding Plan connection over Anthropic Messages and OpenAI Chat while changing only the selected protocol
• run a paired smoke preflight before the remaining paid matrix, validate complete continuation-aware request traces, and retain immutable resumable artifacts
• keep recommendations conservative: require unchanged correctness, a successful pair, complete request telemetry, material improvement, and no regression in the other measured dimensions
• preserve normal multi-version Harbor caches by selecting the newest requested task version
• fail closed when invalid host protocol overrides cross the Harbor host-cell boundary

Stack

• Runtime protocol support: feat(runtime): support Kimi OpenAI protocol #1462 (merged)
• Request-trace evidence: feat(runtime): validate provider request traces #1464 (merged)
• Benchmark harness: this PR, rebuilt as one commit on the current main

Verification

• npm ci
• npm run build:test
• npm run test:dist
• focused launcher and Headless benchmark suites
• npm run typecheck
• npm run lint
• npm run format:check
• git diff --check
• complete local diff review and independent Codex review: no findings
• GitHub CI: typecheck, test, and e2e all passed on 6a9a8b89

No paid/account-plan benchmark was run. The Anthropic-compatible default remains unchanged.

Refs #1269.

[Astro-Han]

I do not think this PR is ready to merge in its current shape. The concern is not the line count by itself: the PR combines three independently owned seams—runtime protocol support, request-trace evidence, and the benchmark harness—and the concrete failures found during review align with those boundaries.

I recommend converting this PR back to draft and splitting it before continuing review.

Blocking findings

Runtime protocol support

• Empty reasoning is not preserved. model-adapter.ts drops reasoning_content: "" / reasoning: "" because it only emits a thinking event when the string is non-empty. Preserved-thinking backends require the field to round-trip even when empty, so a valid tool-call response can make the next request fail.
• The detected OpenAI reasoning dialect has the wrong lifetime. createKimiOpenAiTransport() stores it in a model-instance closure, while AiSdkBackend.send() resolves a new model for every user turn. An endpoint that returns reasoning works for later steps in the first turn, then the next turn starts again with reasoning_content.
• The new product-loop test covers multiple provider steps inside one send(), but not two user turns or an empty reasoning field, so both lifecycle failures remain green.

Request-trace evidence

• requestPayloadWithoutProviderOptionsHash was added as a required field while the capture remains schema version 1. The new reader silently drops valid traces created before this PR; reading a legacy v1 capture now produces captures: [].
• The comparison validates capture/attempt consistency internally, but does not bind the trace’s run/session/turn identity to the task execution that returned the path. The new success fixture itself uses task-specific runtime refs but fixed run-1 / session-1 trace events and still passes.
• Malformed or torn attempt records are silently skipped. If another attempt still references the capture, validation passes and request count, failures, latency, and token usage can be understated.

Benchmark harness and recommendation

• The immutable/resume fingerprint does not cover the full environment inherited by Harbor. Variables such as MAKA_CONTEXT_*, continuation settings, and stream/command timeouts can change between resumes, mixing different runtime policies into one supposedly single-variable comparison.
• The smoke comparison runs only after every paid task, repetition, and arm has completed. A shared request-shape drift therefore consumes the full run before failing instead of acting as a preflight gate.
• The documented kimi-for-coding option is currently guaranteed to fail that smoke check: Anthropic and OpenAI captures expose different SDK-level maxOutputTokens values even though the final wire limit is the same.
• The recommendation gate can produce openai_candidate when both arms fail every task, provided there are enough paired failures to pass the non-inferiority calculation. It also treats any one-metric improvement as sufficient, including a 1 ms latency difference while another metric regresses without bound.
• A terminal event without a trace poisons the immutable run: resume retrieves the same event and throws again instead of producing an invalid/not-cleared report.
• The default Harbor task cache root fails when a task has multiple cached versions, which is a normal local-cache state. The documented dry-run is reproducibly blocked by duplicate task IDs.
• The documented dry-run and paid-run commands do not set a stable run ID, so they validate and execute different manifests by default.

CI is green, and the happy-path product-loop coverage is useful, but the current tests do not establish the core experiment invariants: cross-turn reasoning replay, empty reasoning, legacy trace compatibility, execution identity, resume under environment changes, real cross-protocol captures, or complete ordered tool-call arguments/results.

Suggested split

1. Runtime protocol support

   • Existing Kimi provider protocol selection
   • Model/runtime/factory and OpenAI transport changes
   • Cross-turn reasoning, empty reasoning, tool pairing, and usage contract tests

2. Request-trace evidence

   • Protocol-independent request payload hash
   • Schema/version compatibility
   • Trace identity and fail-closed completeness validation
   • Reader and artifact compatibility tests

3. Benchmark harness

   • Kimi protocol A/B runner and Harbor wiring
   • Immutable manifest/resume behavior
   • Smoke preflight and recommendation policy
   • Launcher tests and documentation

The first two seams can be reviewed as independent flat PRs from main. The benchmark PR can then be stacked on them because it genuinely depends on both. Tests should remain with the behavior they verify rather than being split into separate test-only PRs.

This split is about ownership, verification, and independent rollback—not splitting for size alone. It should also make the remaining fixes substantially easier to reason about and review.

[Nyvo-io]

Rebuilt this PR according to the requested split. The runtime protocol layer is in merged #1462, the request-trace layer is in #1464, and this branch now contains the benchmark harness stacked on #1464 (c1917e8f) rather than the previous combined implementation.

The benchmark findings from the original review are covered: inherited runtime policy is fingerprinted, the paired smoke check runs before the remaining matrix, equivalent output limits are normalized, recommendations require successful comparable evidence and material non-regressing improvement, terminal missing-trace outcomes resume as stable invalid results, multi-version caches select the newest requested task, and dry/paid runs share an explicit immutable run ID. The host-cell protocol boundary also now preserves invalid values so Kimi validation fails closed.

Local full headless: 1394 tests, 1393 passed, 1 Pier-Python skip, 0 failed. Typecheck/lint/format/diff checks pass, the independent Codex review reported no findings, and GitHub typecheck/test/e2e are all green on 8ed7c034.

I am leaving #1451 as draft while #1464 remains its open dependency.

[Nyvo-io]

Ready for re-review at 6a9a8b8. The branch is now one benchmark-only commit on current main after #1462 and #1464 merged. The original harness findings were rechecked against the final diff; local build, full serial workspace tests, typecheck, lint, format, focused benchmark tests, and an independent Codex review all passed with no findings. GitHub typecheck, test, and e2e are green. No paid benchmark was run, and the Anthropic-compatible default is unchanged.

[Astro-Han]

Reviewed 6a9a8b89. The benchmark-only split looks right, but one experiment-validity issue remains:

• [P1] compareKimiProtocolSmokeTrace() checks only the first step === 0 capture and always reports requestCount: 1. Continuation traces can contain several provider requests, so a later request can change its shared history or non-protocol parameters without failing the smoke gate. That can produce an openai_candidate result from a confounded run. Please compare every capture in order, require matching request/step counts, and add a test where a request after the first one differs.

The earlier split, resume, and recommendation-gate concerns otherwise look addressed.

[Nyvo-io]

Ready for re-review at 730c6302. The smoke gate now compares every capture in original order, requires matching request counts and step sequences, checks shared/model/non-protocol evidence per request, and verifies both arms use the same provider connection while differing in provider_options. Continuation regressions cover later-history drift plus count and step mismatches. Local repository build, full serial workspace tests, typecheck, lint, format, diff checks, and independent Codex review all passed; GitHub typecheck, test, and e2e are green. No paid benchmark was run.

[Astro-Han]

Rechecked 730c6302. The earlier P1 is fixed: the smoke gate now compares every provider request in order and fails on count, step, shared-history, model, connection, non-protocol parameter, or provider-options drift. The continuation regressions cover the previously missed later-request case. Focused tests pass 26/26, CI is green, and I found no remaining P0–P3 issues.

[Nyvo-io]

Summary

• keep this PR as the benchmark-harness layer requested in the maintainer split
• compare the existing Kimi Coding Plan connection over Anthropic Messages and OpenAI Chat while changing only the selected protocol
• run a paired smoke preflight before the remaining paid matrix, validate complete continuation-aware request traces, and retain immutable resumable artifacts
• keep recommendations conservative: require unchanged correctness, a successful pair, complete request telemetry, material improvement, and no regression in the other measured dimensions
• preserve normal multi-version Harbor caches by selecting the newest requested task version
• fail closed when invalid host protocol overrides cross the Harbor host-cell boundary

Stack

• Runtime protocol support: feat(runtime): support Kimi OpenAI protocol #1462 (merged)
• Request-trace evidence: feat(runtime): validate provider request traces #1464 (merged)
• Benchmark harness: this PR, rebuilt as one commit on the current main

Verification

• npm ci
• npm run build:test
• npm run test:dist
• focused launcher and Headless benchmark suites
• npm run typecheck
• npm run lint
• npm run format:check
• git diff --check
• complete local diff review and independent Codex review: no findings
• GitHub CI: typecheck, test, and e2e all passed on 6a9a8b89

No paid/account-plan benchmark was run. The Anthropic-compatible default remains unchanged.

Refs #1269.

[Astro-Han]

I do not think this PR is ready to merge in its current shape. The concern is not the line count by itself: the PR combines three independently owned seams—runtime protocol support, request-trace evidence, and the benchmark harness—and the concrete failures found during review align with those boundaries.

I recommend converting this PR back to draft and splitting it before continuing review.

Blocking findings

Runtime protocol support

• Empty reasoning is not preserved. model-adapter.ts drops reasoning_content: "" / reasoning: "" because it only emits a thinking event when the string is non-empty. Preserved-thinking backends require the field to round-trip even when empty, so a valid tool-call response can make the next request fail.
• The detected OpenAI reasoning dialect has the wrong lifetime. createKimiOpenAiTransport() stores it in a model-instance closure, while AiSdkBackend.send() resolves a new model for every user turn. An endpoint that returns reasoning works for later steps in the first turn, then the next turn starts again with reasoning_content.
• The new product-loop test covers multiple provider steps inside one send(), but not two user turns or an empty reasoning field, so both lifecycle failures remain green.

Request-trace evidence

• requestPayloadWithoutProviderOptionsHash was added as a required field while the capture remains schema version 1. The new reader silently drops valid traces created before this PR; reading a legacy v1 capture now produces captures: [].
• The comparison validates capture/attempt consistency internally, but does not bind the trace’s run/session/turn identity to the task execution that returned the path. The new success fixture itself uses task-specific runtime refs but fixed run-1 / session-1 trace events and still passes.
• Malformed or torn attempt records are silently skipped. If another attempt still references the capture, validation passes and request count, failures, latency, and token usage can be understated.

Benchmark harness and recommendation

• The immutable/resume fingerprint does not cover the full environment inherited by Harbor. Variables such as MAKA_CONTEXT_*, continuation settings, and stream/command timeouts can change between resumes, mixing different runtime policies into one supposedly single-variable comparison.
• The smoke comparison runs only after every paid task, repetition, and arm has completed. A shared request-shape drift therefore consumes the full run before failing instead of acting as a preflight gate.
• The documented kimi-for-coding option is currently guaranteed to fail that smoke check: Anthropic and OpenAI captures expose different SDK-level maxOutputTokens values even though the final wire limit is the same.
• The recommendation gate can produce openai_candidate when both arms fail every task, provided there are enough paired failures to pass the non-inferiority calculation. It also treats any one-metric improvement as sufficient, including a 1 ms latency difference while another metric regresses without bound.
• A terminal event without a trace poisons the immutable run: resume retrieves the same event and throws again instead of producing an invalid/not-cleared report.
• The default Harbor task cache root fails when a task has multiple cached versions, which is a normal local-cache state. The documented dry-run is reproducibly blocked by duplicate task IDs.
• The documented dry-run and paid-run commands do not set a stable run ID, so they validate and execute different manifests by default.

CI is green, and the happy-path product-loop coverage is useful, but the current tests do not establish the core experiment invariants: cross-turn reasoning replay, empty reasoning, legacy trace compatibility, execution identity, resume under environment changes, real cross-protocol captures, or complete ordered tool-call arguments/results.

Suggested split

1. Runtime protocol support

   • Existing Kimi provider protocol selection
   • Model/runtime/factory and OpenAI transport changes
   • Cross-turn reasoning, empty reasoning, tool pairing, and usage contract tests

2. Request-trace evidence

   • Protocol-independent request payload hash
   • Schema/version compatibility
   • Trace identity and fail-closed completeness validation
   • Reader and artifact compatibility tests

3. Benchmark harness

   • Kimi protocol A/B runner and Harbor wiring
   • Immutable manifest/resume behavior
   • Smoke preflight and recommendation policy
   • Launcher tests and documentation

The first two seams can be reviewed as independent flat PRs from main. The benchmark PR can then be stacked on them because it genuinely depends on both. Tests should remain with the behavior they verify rather than being split into separate test-only PRs.

This split is about ownership, verification, and independent rollback—not splitting for size alone. It should also make the remaining fixes substantially easier to reason about and review.

[Nyvo-io]

Rebuilt this PR according to the requested split. The runtime protocol layer is in merged #1462, the request-trace layer is in #1464, and this branch now contains the benchmark harness stacked on #1464 (c1917e8f) rather than the previous combined implementation.

The benchmark findings from the original review are covered: inherited runtime policy is fingerprinted, the paired smoke check runs before the remaining matrix, equivalent output limits are normalized, recommendations require successful comparable evidence and material non-regressing improvement, terminal missing-trace outcomes resume as stable invalid results, multi-version caches select the newest requested task, and dry/paid runs share an explicit immutable run ID. The host-cell protocol boundary also now preserves invalid values so Kimi validation fails closed.

Local full headless: 1394 tests, 1393 passed, 1 Pier-Python skip, 0 failed. Typecheck/lint/format/diff checks pass, the independent Codex review reported no findings, and GitHub typecheck/test/e2e are all green on 8ed7c034.

I am leaving #1451 as draft while #1464 remains its open dependency.

[Nyvo-io]

Ready for re-review at 6a9a8b8. The branch is now one benchmark-only commit on current main after #1462 and #1464 merged. The original harness findings were rechecked against the final diff; local build, full serial workspace tests, typecheck, lint, format, focused benchmark tests, and an independent Codex review all passed with no findings. GitHub typecheck, test, and e2e are green. No paid benchmark was run, and the Anthropic-compatible default is unchanged.

[Astro-Han]

Reviewed 6a9a8b89. The benchmark-only split looks right, but one experiment-validity issue remains:

• [P1] compareKimiProtocolSmokeTrace() checks only the first step === 0 capture and always reports requestCount: 1. Continuation traces can contain several provider requests, so a later request can change its shared history or non-protocol parameters without failing the smoke gate. That can produce an openai_candidate result from a confounded run. Please compare every capture in order, require matching request/step counts, and add a test where a request after the first one differs.

The earlier split, resume, and recommendation-gate concerns otherwise look addressed.

[Nyvo-io]

Ready for re-review at 730c6302. The smoke gate now compares every capture in original order, requires matching request counts and step sequences, checks shared/model/non-protocol evidence per request, and verifies both arms use the same provider connection while differing in provider_options. Continuation regressions cover later-history drift plus count and step mismatches. Local repository build, full serial workspace tests, typecheck, lint, format, diff checks, and independent Codex review all passed; GitHub typecheck, test, and e2e are green. No paid benchmark was run.

[Astro-Han]

Rechecked 730c6302. The earlier P1 is fixed: the smoke gate now compares every provider request in order and fails on count, step, shared-history, model, connection, non-protocol parameter, or provider-options drift. The continuation regressions cover the previously missed later-request case. Focused tests pass 26/26, CI is green, and I found no remaining P0–P3 issues.

[Nyvo-io]

Summary

• keep this PR as the benchmark-harness layer requested in the maintainer split
• compare the existing Kimi Coding Plan connection over Anthropic Messages and OpenAI Chat while changing only the selected protocol
• run a paired smoke preflight before the remaining paid matrix, validate complete continuation-aware request traces, and retain immutable resumable artifacts
• keep recommendations conservative: require unchanged correctness, a successful pair, complete request telemetry, material improvement, and no regression in the other measured dimensions
• preserve normal multi-version Harbor caches by selecting the newest requested task version
• fail closed when invalid host protocol overrides cross the Harbor host-cell boundary

Stack

• Runtime protocol support: feat(runtime): support Kimi OpenAI protocol #1462 (merged)
• Request-trace evidence: feat(runtime): validate provider request traces #1464 (merged)
• Benchmark harness: this PR, rebuilt as one commit on the current main

Verification

• npm ci
• npm run build:test
• npm run test:dist
• focused launcher and Headless benchmark suites
• npm run typecheck
• npm run lint
• npm run format:check
• git diff --check
• complete local diff review and independent Codex review: no findings
• GitHub CI: typecheck, test, and e2e all passed on 6a9a8b89

No paid/account-plan benchmark was run. The Anthropic-compatible default remains unchanged.

Refs #1269.

[Astro-Han]

I do not think this PR is ready to merge in its current shape. The concern is not the line count by itself: the PR combines three independently owned seams—runtime protocol support, request-trace evidence, and the benchmark harness—and the concrete failures found during review align with those boundaries.

I recommend converting this PR back to draft and splitting it before continuing review.

Blocking findings

Runtime protocol support

• Empty reasoning is not preserved. model-adapter.ts drops reasoning_content: "" / reasoning: "" because it only emits a thinking event when the string is non-empty. Preserved-thinking backends require the field to round-trip even when empty, so a valid tool-call response can make the next request fail.
• The detected OpenAI reasoning dialect has the wrong lifetime. createKimiOpenAiTransport() stores it in a model-instance closure, while AiSdkBackend.send() resolves a new model for every user turn. An endpoint that returns reasoning works for later steps in the first turn, then the next turn starts again with reasoning_content.
• The new product-loop test covers multiple provider steps inside one send(), but not two user turns or an empty reasoning field, so both lifecycle failures remain green.

Request-trace evidence

• requestPayloadWithoutProviderOptionsHash was added as a required field while the capture remains schema version 1. The new reader silently drops valid traces created before this PR; reading a legacy v1 capture now produces captures: [].
• The comparison validates capture/attempt consistency internally, but does not bind the trace’s run/session/turn identity to the task execution that returned the path. The new success fixture itself uses task-specific runtime refs but fixed run-1 / session-1 trace events and still passes.
• Malformed or torn attempt records are silently skipped. If another attempt still references the capture, validation passes and request count, failures, latency, and token usage can be understated.

Benchmark harness and recommendation

• The immutable/resume fingerprint does not cover the full environment inherited by Harbor. Variables such as MAKA_CONTEXT_*, continuation settings, and stream/command timeouts can change between resumes, mixing different runtime policies into one supposedly single-variable comparison.
• The smoke comparison runs only after every paid task, repetition, and arm has completed. A shared request-shape drift therefore consumes the full run before failing instead of acting as a preflight gate.
• The documented kimi-for-coding option is currently guaranteed to fail that smoke check: Anthropic and OpenAI captures expose different SDK-level maxOutputTokens values even though the final wire limit is the same.
• The recommendation gate can produce openai_candidate when both arms fail every task, provided there are enough paired failures to pass the non-inferiority calculation. It also treats any one-metric improvement as sufficient, including a 1 ms latency difference while another metric regresses without bound.
• A terminal event without a trace poisons the immutable run: resume retrieves the same event and throws again instead of producing an invalid/not-cleared report.
• The default Harbor task cache root fails when a task has multiple cached versions, which is a normal local-cache state. The documented dry-run is reproducibly blocked by duplicate task IDs.
• The documented dry-run and paid-run commands do not set a stable run ID, so they validate and execute different manifests by default.

CI is green, and the happy-path product-loop coverage is useful, but the current tests do not establish the core experiment invariants: cross-turn reasoning replay, empty reasoning, legacy trace compatibility, execution identity, resume under environment changes, real cross-protocol captures, or complete ordered tool-call arguments/results.

Suggested split

1. Runtime protocol support

   • Existing Kimi provider protocol selection
   • Model/runtime/factory and OpenAI transport changes
   • Cross-turn reasoning, empty reasoning, tool pairing, and usage contract tests

2. Request-trace evidence

   • Protocol-independent request payload hash
   • Schema/version compatibility
   • Trace identity and fail-closed completeness validation
   • Reader and artifact compatibility tests

3. Benchmark harness

   • Kimi protocol A/B runner and Harbor wiring
   • Immutable manifest/resume behavior
   • Smoke preflight and recommendation policy
   • Launcher tests and documentation

The first two seams can be reviewed as independent flat PRs from main. The benchmark PR can then be stacked on them because it genuinely depends on both. Tests should remain with the behavior they verify rather than being split into separate test-only PRs.

This split is about ownership, verification, and independent rollback—not splitting for size alone. It should also make the remaining fixes substantially easier to reason about and review.

[Nyvo-io]

Rebuilt this PR according to the requested split. The runtime protocol layer is in merged #1462, the request-trace layer is in #1464, and this branch now contains the benchmark harness stacked on #1464 (c1917e8f) rather than the previous combined implementation.

The benchmark findings from the original review are covered: inherited runtime policy is fingerprinted, the paired smoke check runs before the remaining matrix, equivalent output limits are normalized, recommendations require successful comparable evidence and material non-regressing improvement, terminal missing-trace outcomes resume as stable invalid results, multi-version caches select the newest requested task, and dry/paid runs share an explicit immutable run ID. The host-cell protocol boundary also now preserves invalid values so Kimi validation fails closed.

Local full headless: 1394 tests, 1393 passed, 1 Pier-Python skip, 0 failed. Typecheck/lint/format/diff checks pass, the independent Codex review reported no findings, and GitHub typecheck/test/e2e are all green on 8ed7c034.

I am leaving #1451 as draft while #1464 remains its open dependency.

[Nyvo-io]

Ready for re-review at 6a9a8b8. The branch is now one benchmark-only commit on current main after #1462 and #1464 merged. The original harness findings were rechecked against the final diff; local build, full serial workspace tests, typecheck, lint, format, focused benchmark tests, and an independent Codex review all passed with no findings. GitHub typecheck, test, and e2e are green. No paid benchmark was run, and the Anthropic-compatible default is unchanged.

[Astro-Han]

Reviewed 6a9a8b89. The benchmark-only split looks right, but one experiment-validity issue remains:

• [P1] compareKimiProtocolSmokeTrace() checks only the first step === 0 capture and always reports requestCount: 1. Continuation traces can contain several provider requests, so a later request can change its shared history or non-protocol parameters without failing the smoke gate. That can produce an openai_candidate result from a confounded run. Please compare every capture in order, require matching request/step counts, and add a test where a request after the first one differs.

The earlier split, resume, and recommendation-gate concerns otherwise look addressed.

[Nyvo-io]

Ready for re-review at 730c6302. The smoke gate now compares every capture in original order, requires matching request counts and step sequences, checks shared/model/non-protocol evidence per request, and verifies both arms use the same provider connection while differing in provider_options. Continuation regressions cover later-history drift plus count and step mismatches. Local repository build, full serial workspace tests, typecheck, lint, format, diff checks, and independent Codex review all passed; GitHub typecheck, test, and e2e are green. No paid benchmark was run.

[Astro-Han]

Rechecked 730c6302. The earlier P1 is fixed: the smoke gate now compares every provider request in order and fails on count, step, shared-history, model, connection, non-protocol parameter, or provider-options drift. The continuation regressions cover the previously missed later-request case. Focused tests pass 26/26, CI is green, and I found no remaining P0–P3 issues.

[Nyvo-io]

Summary

• keep this PR as the benchmark-harness layer requested in the maintainer split
• compare the existing Kimi Coding Plan connection over Anthropic Messages and OpenAI Chat while changing only the selected protocol
• run a paired smoke preflight before the remaining paid matrix, validate complete continuation-aware request traces, and retain immutable resumable artifacts
• keep recommendations conservative: require unchanged correctness, a successful pair, complete request telemetry, material improvement, and no regression in the other measured dimensions
• preserve normal multi-version Harbor caches by selecting the newest requested task version
• fail closed when invalid host protocol overrides cross the Harbor host-cell boundary

Stack

• Runtime protocol support: feat(runtime): support Kimi OpenAI protocol #1462 (merged)
• Request-trace evidence: feat(runtime): validate provider request traces #1464 (merged)
• Benchmark harness: this PR, rebuilt as one commit on the current main

Verification

• npm ci
• npm run build:test
• npm run test:dist
• focused launcher and Headless benchmark suites
• npm run typecheck
• npm run lint
• npm run format:check
• git diff --check
• complete local diff review and independent Codex review: no findings
• GitHub CI: typecheck, test, and e2e all passed on 6a9a8b89

No paid/account-plan benchmark was run. The Anthropic-compatible default remains unchanged.

Refs #1269.

[Astro-Han]

I do not think this PR is ready to merge in its current shape. The concern is not the line count by itself: the PR combines three independently owned seams—runtime protocol support, request-trace evidence, and the benchmark harness—and the concrete failures found during review align with those boundaries.

I recommend converting this PR back to draft and splitting it before continuing review.

Blocking findings

Runtime protocol support

• Empty reasoning is not preserved. model-adapter.ts drops reasoning_content: "" / reasoning: "" because it only emits a thinking event when the string is non-empty. Preserved-thinking backends require the field to round-trip even when empty, so a valid tool-call response can make the next request fail.
• The detected OpenAI reasoning dialect has the wrong lifetime. createKimiOpenAiTransport() stores it in a model-instance closure, while AiSdkBackend.send() resolves a new model for every user turn. An endpoint that returns reasoning works for later steps in the first turn, then the next turn starts again with reasoning_content.
• The new product-loop test covers multiple provider steps inside one send(), but not two user turns or an empty reasoning field, so both lifecycle failures remain green.

Request-trace evidence

• requestPayloadWithoutProviderOptionsHash was added as a required field while the capture remains schema version 1. The new reader silently drops valid traces created before this PR; reading a legacy v1 capture now produces captures: [].
• The comparison validates capture/attempt consistency internally, but does not bind the trace’s run/session/turn identity to the task execution that returned the path. The new success fixture itself uses task-specific runtime refs but fixed run-1 / session-1 trace events and still passes.
• Malformed or torn attempt records are silently skipped. If another attempt still references the capture, validation passes and request count, failures, latency, and token usage can be understated.

Benchmark harness and recommendation

• The immutable/resume fingerprint does not cover the full environment inherited by Harbor. Variables such as MAKA_CONTEXT_*, continuation settings, and stream/command timeouts can change between resumes, mixing different runtime policies into one supposedly single-variable comparison.
• The smoke comparison runs only after every paid task, repetition, and arm has completed. A shared request-shape drift therefore consumes the full run before failing instead of acting as a preflight gate.
• The documented kimi-for-coding option is currently guaranteed to fail that smoke check: Anthropic and OpenAI captures expose different SDK-level maxOutputTokens values even though the final wire limit is the same.
• The recommendation gate can produce openai_candidate when both arms fail every task, provided there are enough paired failures to pass the non-inferiority calculation. It also treats any one-metric improvement as sufficient, including a 1 ms latency difference while another metric regresses without bound.
• A terminal event without a trace poisons the immutable run: resume retrieves the same event and throws again instead of producing an invalid/not-cleared report.
• The default Harbor task cache root fails when a task has multiple cached versions, which is a normal local-cache state. The documented dry-run is reproducibly blocked by duplicate task IDs.
• The documented dry-run and paid-run commands do not set a stable run ID, so they validate and execute different manifests by default.

CI is green, and the happy-path product-loop coverage is useful, but the current tests do not establish the core experiment invariants: cross-turn reasoning replay, empty reasoning, legacy trace compatibility, execution identity, resume under environment changes, real cross-protocol captures, or complete ordered tool-call arguments/results.

Suggested split

1. Runtime protocol support

   • Existing Kimi provider protocol selection
   • Model/runtime/factory and OpenAI transport changes
   • Cross-turn reasoning, empty reasoning, tool pairing, and usage contract tests

2. Request-trace evidence

   • Protocol-independent request payload hash
   • Schema/version compatibility
   • Trace identity and fail-closed completeness validation
   • Reader and artifact compatibility tests

3. Benchmark harness

   • Kimi protocol A/B runner and Harbor wiring
   • Immutable manifest/resume behavior
   • Smoke preflight and recommendation policy
   • Launcher tests and documentation

The first two seams can be reviewed as independent flat PRs from main. The benchmark PR can then be stacked on them because it genuinely depends on both. Tests should remain with the behavior they verify rather than being split into separate test-only PRs.

This split is about ownership, verification, and independent rollback—not splitting for size alone. It should also make the remaining fixes substantially easier to reason about and review.

[Nyvo-io]

Rebuilt this PR according to the requested split. The runtime protocol layer is in merged #1462, the request-trace layer is in #1464, and this branch now contains the benchmark harness stacked on #1464 (c1917e8f) rather than the previous combined implementation.

The benchmark findings from the original review are covered: inherited runtime policy is fingerprinted, the paired smoke check runs before the remaining matrix, equivalent output limits are normalized, recommendations require successful comparable evidence and material non-regressing improvement, terminal missing-trace outcomes resume as stable invalid results, multi-version caches select the newest requested task, and dry/paid runs share an explicit immutable run ID. The host-cell protocol boundary also now preserves invalid values so Kimi validation fails closed.

Local full headless: 1394 tests, 1393 passed, 1 Pier-Python skip, 0 failed. Typecheck/lint/format/diff checks pass, the independent Codex review reported no findings, and GitHub typecheck/test/e2e are all green on 8ed7c034.

I am leaving #1451 as draft while #1464 remains its open dependency.

[Nyvo-io]

Ready for re-review at 6a9a8b8. The branch is now one benchmark-only commit on current main after #1462 and #1464 merged. The original harness findings were rechecked against the final diff; local build, full serial workspace tests, typecheck, lint, format, focused benchmark tests, and an independent Codex review all passed with no findings. GitHub typecheck, test, and e2e are green. No paid benchmark was run, and the Anthropic-compatible default is unchanged.

[Astro-Han]

Reviewed 6a9a8b89. The benchmark-only split looks right, but one experiment-validity issue remains:

• [P1] compareKimiProtocolSmokeTrace() checks only the first step === 0 capture and always reports requestCount: 1. Continuation traces can contain several provider requests, so a later request can change its shared history or non-protocol parameters without failing the smoke gate. That can produce an openai_candidate result from a confounded run. Please compare every capture in order, require matching request/step counts, and add a test where a request after the first one differs.

The earlier split, resume, and recommendation-gate concerns otherwise look addressed.

[Nyvo-io]

Ready for re-review at 730c6302. The smoke gate now compares every capture in original order, requires matching request counts and step sequences, checks shared/model/non-protocol evidence per request, and verifies both arms use the same provider connection while differing in provider_options. Continuation regressions cover later-history drift plus count and step mismatches. Local repository build, full serial workspace tests, typecheck, lint, format, diff checks, and independent Codex review all passed; GitHub typecheck, test, and e2e are green. No paid benchmark was run.

[Astro-Han]

Rechecked 730c6302. The earlier P1 is fixed: the smoke gate now compares every provider request in order and fails on count, step, shared-history, model, connection, non-protocol parameter, or provider-options drift. The continuation regressions cover the previously missed later-request case. Focused tests pass 26/26, CI is green, and I found no remaining P0–P3 issues.

[Nyvo-io]

Summary

• keep this PR as the benchmark-harness layer requested in the maintainer split
• compare the existing Kimi Coding Plan connection over Anthropic Messages and OpenAI Chat while changing only the selected protocol
• run a paired smoke preflight before the remaining paid matrix, validate complete continuation-aware request traces, and retain immutable resumable artifacts
• keep recommendations conservative: require unchanged correctness, a successful pair, complete request telemetry, material improvement, and no regression in the other measured dimensions
• preserve normal multi-version Harbor caches by selecting the newest requested task version
• fail closed when invalid host protocol overrides cross the Harbor host-cell boundary

Stack

• Runtime protocol support: feat(runtime): support Kimi OpenAI protocol #1462 (merged)
• Request-trace evidence: feat(runtime): validate provider request traces #1464 (merged)
• Benchmark harness: this PR, rebuilt as one commit on the current main

Verification

• npm ci
• npm run build:test
• npm run test:dist
• focused launcher and Headless benchmark suites
• npm run typecheck
• npm run lint
• npm run format:check
• git diff --check
• complete local diff review and independent Codex review: no findings
• GitHub CI: typecheck, test, and e2e all passed on 6a9a8b89

No paid/account-plan benchmark was run. The Anthropic-compatible default remains unchanged.

Refs #1269.

[Astro-Han]

I do not think this PR is ready to merge in its current shape. The concern is not the line count by itself: the PR combines three independently owned seams—runtime protocol support, request-trace evidence, and the benchmark harness—and the concrete failures found during review align with those boundaries.

I recommend converting this PR back to draft and splitting it before continuing review.

Blocking findings

Runtime protocol support

• Empty reasoning is not preserved. model-adapter.ts drops reasoning_content: "" / reasoning: "" because it only emits a thinking event when the string is non-empty. Preserved-thinking backends require the field to round-trip even when empty, so a valid tool-call response can make the next request fail.
• The detected OpenAI reasoning dialect has the wrong lifetime. createKimiOpenAiTransport() stores it in a model-instance closure, while AiSdkBackend.send() resolves a new model for every user turn. An endpoint that returns reasoning works for later steps in the first turn, then the next turn starts again with reasoning_content.
• The new product-loop test covers multiple provider steps inside one send(), but not two user turns or an empty reasoning field, so both lifecycle failures remain green.

Request-trace evidence

• requestPayloadWithoutProviderOptionsHash was added as a required field while the capture remains schema version 1. The new reader silently drops valid traces created before this PR; reading a legacy v1 capture now produces captures: [].
• The comparison validates capture/attempt consistency internally, but does not bind the trace’s run/session/turn identity to the task execution that returned the path. The new success fixture itself uses task-specific runtime refs but fixed run-1 / session-1 trace events and still passes.
• Malformed or torn attempt records are silently skipped. If another attempt still references the capture, validation passes and request count, failures, latency, and token usage can be understated.

Benchmark harness and recommendation

• The immutable/resume fingerprint does not cover the full environment inherited by Harbor. Variables such as MAKA_CONTEXT_*, continuation settings, and stream/command timeouts can change between resumes, mixing different runtime policies into one supposedly single-variable comparison.
• The smoke comparison runs only after every paid task, repetition, and arm has completed. A shared request-shape drift therefore consumes the full run before failing instead of acting as a preflight gate.
• The documented kimi-for-coding option is currently guaranteed to fail that smoke check: Anthropic and OpenAI captures expose different SDK-level maxOutputTokens values even though the final wire limit is the same.
• The recommendation gate can produce openai_candidate when both arms fail every task, provided there are enough paired failures to pass the non-inferiority calculation. It also treats any one-metric improvement as sufficient, including a 1 ms latency difference while another metric regresses without bound.
• A terminal event without a trace poisons the immutable run: resume retrieves the same event and throws again instead of producing an invalid/not-cleared report.
• The default Harbor task cache root fails when a task has multiple cached versions, which is a normal local-cache state. The documented dry-run is reproducibly blocked by duplicate task IDs.
• The documented dry-run and paid-run commands do not set a stable run ID, so they validate and execute different manifests by default.

CI is green, and the happy-path product-loop coverage is useful, but the current tests do not establish the core experiment invariants: cross-turn reasoning replay, empty reasoning, legacy trace compatibility, execution identity, resume under environment changes, real cross-protocol captures, or complete ordered tool-call arguments/results.

Suggested split

1. Runtime protocol support

   • Existing Kimi provider protocol selection
   • Model/runtime/factory and OpenAI transport changes
   • Cross-turn reasoning, empty reasoning, tool pairing, and usage contract tests

2. Request-trace evidence

   • Protocol-independent request payload hash
   • Schema/version compatibility
   • Trace identity and fail-closed completeness validation
   • Reader and artifact compatibility tests

3. Benchmark harness

   • Kimi protocol A/B runner and Harbor wiring
   • Immutable manifest/resume behavior
   • Smoke preflight and recommendation policy
   • Launcher tests and documentation

The first two seams can be reviewed as independent flat PRs from main. The benchmark PR can then be stacked on them because it genuinely depends on both. Tests should remain with the behavior they verify rather than being split into separate test-only PRs.

This split is about ownership, verification, and independent rollback—not splitting for size alone. It should also make the remaining fixes substantially easier to reason about and review.

[Nyvo-io]

Rebuilt this PR according to the requested split. The runtime protocol layer is in merged #1462, the request-trace layer is in #1464, and this branch now contains the benchmark harness stacked on #1464 (c1917e8f) rather than the previous combined implementation.

The benchmark findings from the original review are covered: inherited runtime policy is fingerprinted, the paired smoke check runs before the remaining matrix, equivalent output limits are normalized, recommendations require successful comparable evidence and material non-regressing improvement, terminal missing-trace outcomes resume as stable invalid results, multi-version caches select the newest requested task, and dry/paid runs share an explicit immutable run ID. The host-cell protocol boundary also now preserves invalid values so Kimi validation fails closed.

Local full headless: 1394 tests, 1393 passed, 1 Pier-Python skip, 0 failed. Typecheck/lint/format/diff checks pass, the independent Codex review reported no findings, and GitHub typecheck/test/e2e are all green on 8ed7c034.

I am leaving #1451 as draft while #1464 remains its open dependency.

[Nyvo-io]

Ready for re-review at 6a9a8b8. The branch is now one benchmark-only commit on current main after #1462 and #1464 merged. The original harness findings were rechecked against the final diff; local build, full serial workspace tests, typecheck, lint, format, focused benchmark tests, and an independent Codex review all passed with no findings. GitHub typecheck, test, and e2e are green. No paid benchmark was run, and the Anthropic-compatible default is unchanged.

[Astro-Han]

Reviewed 6a9a8b89. The benchmark-only split looks right, but one experiment-validity issue remains:

• [P1] compareKimiProtocolSmokeTrace() checks only the first step === 0 capture and always reports requestCount: 1. Continuation traces can contain several provider requests, so a later request can change its shared history or non-protocol parameters without failing the smoke gate. That can produce an openai_candidate result from a confounded run. Please compare every capture in order, require matching request/step counts, and add a test where a request after the first one differs.

The earlier split, resume, and recommendation-gate concerns otherwise look addressed.

[Nyvo-io]

Ready for re-review at 730c6302. The smoke gate now compares every capture in original order, requires matching request counts and step sequences, checks shared/model/non-protocol evidence per request, and verifies both arms use the same provider connection while differing in provider_options. Continuation regressions cover later-history drift plus count and step mismatches. Local repository build, full serial workspace tests, typecheck, lint, format, diff checks, and independent Codex review all passed; GitHub typecheck, test, and e2e are green. No paid benchmark was run.

[Astro-Han]

Rechecked 730c6302. The earlier P1 is fixed: the smoke gate now compares every provider request in order and fails on count, step, shared-history, model, connection, non-protocol parameter, or provider-options drift. The continuation regressions cover the previously missed later-request case. Focused tests pass 26/26, CI is green, and I found no remaining P0–P3 issues.

[Nyvo-io]

Summary

• keep this PR as the benchmark-harness layer requested in the maintainer split
• compare the existing Kimi Coding Plan connection over Anthropic Messages and OpenAI Chat while changing only the selected protocol
• run a paired smoke preflight before the remaining paid matrix, validate complete continuation-aware request traces, and retain immutable resumable artifacts
• keep recommendations conservative: require unchanged correctness, a successful pair, complete request telemetry, material improvement, and no regression in the other measured dimensions
• preserve normal multi-version Harbor caches by selecting the newest requested task version
• fail closed when invalid host protocol overrides cross the Harbor host-cell boundary

Stack

• Runtime protocol support: feat(runtime): support Kimi OpenAI protocol #1462 (merged)
• Request-trace evidence: feat(runtime): validate provider request traces #1464 (merged)
• Benchmark harness: this PR, rebuilt as one commit on the current main

Verification

• npm ci
• npm run build:test
• npm run test:dist
• focused launcher and Headless benchmark suites
• npm run typecheck
• npm run lint
• npm run format:check
• git diff --check
• complete local diff review and independent Codex review: no findings
• GitHub CI: typecheck, test, and e2e all passed on 6a9a8b89

No paid/account-plan benchmark was run. The Anthropic-compatible default remains unchanged.

Refs #1269.

[Astro-Han]

I do not think this PR is ready to merge in its current shape. The concern is not the line count by itself: the PR combines three independently owned seams—runtime protocol support, request-trace evidence, and the benchmark harness—and the concrete failures found during review align with those boundaries.

I recommend converting this PR back to draft and splitting it before continuing review.

Blocking findings

Runtime protocol support

• Empty reasoning is not preserved. model-adapter.ts drops reasoning_content: "" / reasoning: "" because it only emits a thinking event when the string is non-empty. Preserved-thinking backends require the field to round-trip even when empty, so a valid tool-call response can make the next request fail.
• The detected OpenAI reasoning dialect has the wrong lifetime. createKimiOpenAiTransport() stores it in a model-instance closure, while AiSdkBackend.send() resolves a new model for every user turn. An endpoint that returns reasoning works for later steps in the first turn, then the next turn starts again with reasoning_content.
• The new product-loop test covers multiple provider steps inside one send(), but not two user turns or an empty reasoning field, so both lifecycle failures remain green.

Request-trace evidence

• requestPayloadWithoutProviderOptionsHash was added as a required field while the capture remains schema version 1. The new reader silently drops valid traces created before this PR; reading a legacy v1 capture now produces captures: [].
• The comparison validates capture/attempt consistency internally, but does not bind the trace’s run/session/turn identity to the task execution that returned the path. The new success fixture itself uses task-specific runtime refs but fixed run-1 / session-1 trace events and still passes.
• Malformed or torn attempt records are silently skipped. If another attempt still references the capture, validation passes and request count, failures, latency, and token usage can be understated.

Benchmark harness and recommendation

• The immutable/resume fingerprint does not cover the full environment inherited by Harbor. Variables such as MAKA_CONTEXT_*, continuation settings, and stream/command timeouts can change between resumes, mixing different runtime policies into one supposedly single-variable comparison.
• The smoke comparison runs only after every paid task, repetition, and arm has completed. A shared request-shape drift therefore consumes the full run before failing instead of acting as a preflight gate.
• The documented kimi-for-coding option is currently guaranteed to fail that smoke check: Anthropic and OpenAI captures expose different SDK-level maxOutputTokens values even though the final wire limit is the same.
• The recommendation gate can produce openai_candidate when both arms fail every task, provided there are enough paired failures to pass the non-inferiority calculation. It also treats any one-metric improvement as sufficient, including a 1 ms latency difference while another metric regresses without bound.
• A terminal event without a trace poisons the immutable run: resume retrieves the same event and throws again instead of producing an invalid/not-cleared report.
• The default Harbor task cache root fails when a task has multiple cached versions, which is a normal local-cache state. The documented dry-run is reproducibly blocked by duplicate task IDs.
• The documented dry-run and paid-run commands do not set a stable run ID, so they validate and execute different manifests by default.

CI is green, and the happy-path product-loop coverage is useful, but the current tests do not establish the core experiment invariants: cross-turn reasoning replay, empty reasoning, legacy trace compatibility, execution identity, resume under environment changes, real cross-protocol captures, or complete ordered tool-call arguments/results.

Suggested split

1. Runtime protocol support

   • Existing Kimi provider protocol selection
   • Model/runtime/factory and OpenAI transport changes
   • Cross-turn reasoning, empty reasoning, tool pairing, and usage contract tests

2. Request-trace evidence

   • Protocol-independent request payload hash
   • Schema/version compatibility
   • Trace identity and fail-closed completeness validation
   • Reader and artifact compatibility tests

3. Benchmark harness

   • Kimi protocol A/B runner and Harbor wiring
   • Immutable manifest/resume behavior
   • Smoke preflight and recommendation policy
   • Launcher tests and documentation

The first two seams can be reviewed as independent flat PRs from main. The benchmark PR can then be stacked on them because it genuinely depends on both. Tests should remain with the behavior they verify rather than being split into separate test-only PRs.

This split is about ownership, verification, and independent rollback—not splitting for size alone. It should also make the remaining fixes substantially easier to reason about and review.

[Nyvo-io]

Rebuilt this PR according to the requested split. The runtime protocol layer is in merged #1462, the request-trace layer is in #1464, and this branch now contains the benchmark harness stacked on #1464 (c1917e8f) rather than the previous combined implementation.

The benchmark findings from the original review are covered: inherited runtime policy is fingerprinted, the paired smoke check runs before the remaining matrix, equivalent output limits are normalized, recommendations require successful comparable evidence and material non-regressing improvement, terminal missing-trace outcomes resume as stable invalid results, multi-version caches select the newest requested task, and dry/paid runs share an explicit immutable run ID. The host-cell protocol boundary also now preserves invalid values so Kimi validation fails closed.

Local full headless: 1394 tests, 1393 passed, 1 Pier-Python skip, 0 failed. Typecheck/lint/format/diff checks pass, the independent Codex review reported no findings, and GitHub typecheck/test/e2e are all green on 8ed7c034.

I am leaving #1451 as draft while #1464 remains its open dependency.

[Nyvo-io]

Ready for re-review at 6a9a8b8. The branch is now one benchmark-only commit on current main after #1462 and #1464 merged. The original harness findings were rechecked against the final diff; local build, full serial workspace tests, typecheck, lint, format, focused benchmark tests, and an independent Codex review all passed with no findings. GitHub typecheck, test, and e2e are green. No paid benchmark was run, and the Anthropic-compatible default is unchanged.

[Astro-Han]

Reviewed 6a9a8b89. The benchmark-only split looks right, but one experiment-validity issue remains:

• [P1] compareKimiProtocolSmokeTrace() checks only the first step === 0 capture and always reports requestCount: 1. Continuation traces can contain several provider requests, so a later request can change its shared history or non-protocol parameters without failing the smoke gate. That can produce an openai_candidate result from a confounded run. Please compare every capture in order, require matching request/step counts, and add a test where a request after the first one differs.

The earlier split, resume, and recommendation-gate concerns otherwise look addressed.

[Nyvo-io]

Ready for re-review at 730c6302. The smoke gate now compares every capture in original order, requires matching request counts and step sequences, checks shared/model/non-protocol evidence per request, and verifies both arms use the same provider connection while differing in provider_options. Continuation regressions cover later-history drift plus count and step mismatches. Local repository build, full serial workspace tests, typecheck, lint, format, diff checks, and independent Codex review all passed; GitHub typecheck, test, and e2e are green. No paid benchmark was run.

[Astro-Han]

Rechecked 730c6302. The earlier P1 is fixed: the smoke gate now compares every provider request in order and fails on count, step, shared-history, model, connection, non-protocol parameter, or provider-options drift. The continuation regressions cover the previously missed later-request case. Focused tests pass 26/26, CI is green, and I found no remaining P0–P3 issues.

[Nyvo-io]

Summary

• keep this PR as the benchmark-harness layer requested in the maintainer split
• compare the existing Kimi Coding Plan connection over Anthropic Messages and OpenAI Chat while changing only the selected protocol
• run a paired smoke preflight before the remaining paid matrix, validate complete continuation-aware request traces, and retain immutable resumable artifacts
• keep recommendations conservative: require unchanged correctness, a successful pair, complete request telemetry, material improvement, and no regression in the other measured dimensions
• preserve normal multi-version Harbor caches by selecting the newest requested task version
• fail closed when invalid host protocol overrides cross the Harbor host-cell boundary

Stack

• Runtime protocol support: feat(runtime): support Kimi OpenAI protocol #1462 (merged)
• Request-trace evidence: feat(runtime): validate provider request traces #1464 (merged)
• Benchmark harness: this PR, rebuilt as one commit on the current main

Verification

• npm ci
• npm run build:test
• npm run test:dist
• focused launcher and Headless benchmark suites
• npm run typecheck
• npm run lint
• npm run format:check
• git diff --check
• complete local diff review and independent Codex review: no findings
• GitHub CI: typecheck, test, and e2e all passed on 6a9a8b89

No paid/account-plan benchmark was run. The Anthropic-compatible default remains unchanged.

Refs #1269.

[Astro-Han]

I do not think this PR is ready to merge in its current shape. The concern is not the line count by itself: the PR combines three independently owned seams—runtime protocol support, request-trace evidence, and the benchmark harness—and the concrete failures found during review align with those boundaries.

I recommend converting this PR back to draft and splitting it before continuing review.

Blocking findings

Runtime protocol support

• Empty reasoning is not preserved. model-adapter.ts drops reasoning_content: "" / reasoning: "" because it only emits a thinking event when the string is non-empty. Preserved-thinking backends require the field to round-trip even when empty, so a valid tool-call response can make the next request fail.
• The detected OpenAI reasoning dialect has the wrong lifetime. createKimiOpenAiTransport() stores it in a model-instance closure, while AiSdkBackend.send() resolves a new model for every user turn. An endpoint that returns reasoning works for later steps in the first turn, then the next turn starts again with reasoning_content.
• The new product-loop test covers multiple provider steps inside one send(), but not two user turns or an empty reasoning field, so both lifecycle failures remain green.

Request-trace evidence

• requestPayloadWithoutProviderOptionsHash was added as a required field while the capture remains schema version 1. The new reader silently drops valid traces created before this PR; reading a legacy v1 capture now produces captures: [].
• The comparison validates capture/attempt consistency internally, but does not bind the trace’s run/session/turn identity to the task execution that returned the path. The new success fixture itself uses task-specific runtime refs but fixed run-1 / session-1 trace events and still passes.
• Malformed or torn attempt records are silently skipped. If another attempt still references the capture, validation passes and request count, failures, latency, and token usage can be understated.

Benchmark harness and recommendation

• The immutable/resume fingerprint does not cover the full environment inherited by Harbor. Variables such as MAKA_CONTEXT_*, continuation settings, and stream/command timeouts can change between resumes, mixing different runtime policies into one supposedly single-variable comparison.
• The smoke comparison runs only after every paid task, repetition, and arm has completed. A shared request-shape drift therefore consumes the full run before failing instead of acting as a preflight gate.
• The documented kimi-for-coding option is currently guaranteed to fail that smoke check: Anthropic and OpenAI captures expose different SDK-level maxOutputTokens values even though the final wire limit is the same.
• The recommendation gate can produce openai_candidate when both arms fail every task, provided there are enough paired failures to pass the non-inferiority calculation. It also treats any one-metric improvement as sufficient, including a 1 ms latency difference while another metric regresses without bound.
• A terminal event without a trace poisons the immutable run: resume retrieves the same event and throws again instead of producing an invalid/not-cleared report.
• The default Harbor task cache root fails when a task has multiple cached versions, which is a normal local-cache state. The documented dry-run is reproducibly blocked by duplicate task IDs.
• The documented dry-run and paid-run commands do not set a stable run ID, so they validate and execute different manifests by default.

CI is green, and the happy-path product-loop coverage is useful, but the current tests do not establish the core experiment invariants: cross-turn reasoning replay, empty reasoning, legacy trace compatibility, execution identity, resume under environment changes, real cross-protocol captures, or complete ordered tool-call arguments/results.

Suggested split

1. Runtime protocol support

   • Existing Kimi provider protocol selection
   • Model/runtime/factory and OpenAI transport changes
   • Cross-turn reasoning, empty reasoning, tool pairing, and usage contract tests

2. Request-trace evidence

   • Protocol-independent request payload hash
   • Schema/version compatibility
   • Trace identity and fail-closed completeness validation
   • Reader and artifact compatibility tests

3. Benchmark harness

   • Kimi protocol A/B runner and Harbor wiring
   • Immutable manifest/resume behavior
   • Smoke preflight and recommendation policy
   • Launcher tests and documentation

The first two seams can be reviewed as independent flat PRs from main. The benchmark PR can then be stacked on them because it genuinely depends on both. Tests should remain with the behavior they verify rather than being split into separate test-only PRs.

This split is about ownership, verification, and independent rollback—not splitting for size alone. It should also make the remaining fixes substantially easier to reason about and review.

[Nyvo-io]

Rebuilt this PR according to the requested split. The runtime protocol layer is in merged #1462, the request-trace layer is in #1464, and this branch now contains the benchmark harness stacked on #1464 (c1917e8f) rather than the previous combined implementation.

The benchmark findings from the original review are covered: inherited runtime policy is fingerprinted, the paired smoke check runs before the remaining matrix, equivalent output limits are normalized, recommendations require successful comparable evidence and material non-regressing improvement, terminal missing-trace outcomes resume as stable invalid results, multi-version caches select the newest requested task, and dry/paid runs share an explicit immutable run ID. The host-cell protocol boundary also now preserves invalid values so Kimi validation fails closed.

Local full headless: 1394 tests, 1393 passed, 1 Pier-Python skip, 0 failed. Typecheck/lint/format/diff checks pass, the independent Codex review reported no findings, and GitHub typecheck/test/e2e are all green on 8ed7c034.

I am leaving #1451 as draft while #1464 remains its open dependency.

[Nyvo-io]

Ready for re-review at 6a9a8b8. The branch is now one benchmark-only commit on current main after #1462 and #1464 merged. The original harness findings were rechecked against the final diff; local build, full serial workspace tests, typecheck, lint, format, focused benchmark tests, and an independent Codex review all passed with no findings. GitHub typecheck, test, and e2e are green. No paid benchmark was run, and the Anthropic-compatible default is unchanged.

[Astro-Han]

Reviewed 6a9a8b89. The benchmark-only split looks right, but one experiment-validity issue remains:

• [P1] compareKimiProtocolSmokeTrace() checks only the first step === 0 capture and always reports requestCount: 1. Continuation traces can contain several provider requests, so a later request can change its shared history or non-protocol parameters without failing the smoke gate. That can produce an openai_candidate result from a confounded run. Please compare every capture in order, require matching request/step counts, and add a test where a request after the first one differs.

The earlier split, resume, and recommendation-gate concerns otherwise look addressed.

[Nyvo-io]

Ready for re-review at 730c6302. The smoke gate now compares every capture in original order, requires matching request counts and step sequences, checks shared/model/non-protocol evidence per request, and verifies both arms use the same provider connection while differing in provider_options. Continuation regressions cover later-history drift plus count and step mismatches. Local repository build, full serial workspace tests, typecheck, lint, format, diff checks, and independent Codex review all passed; GitHub typecheck, test, and e2e are green. No paid benchmark was run.

[Astro-Han]

Rechecked 730c6302. The earlier P1 is fixed: the smoke gate now compares every provider request in order and fails on count, step, shared-history, model, connection, non-protocol parameter, or provider-options drift. The continuation regressions cover the previously missed later-request case. Focused tests pass 26/26, CI is green, and I found no remaining P0–P3 issues.

[Nyvo-io]

Summary

• keep this PR as the benchmark-harness layer requested in the maintainer split
• compare the existing Kimi Coding Plan connection over Anthropic Messages and OpenAI Chat while changing only the selected protocol
• run a paired smoke preflight before the remaining paid matrix, validate complete continuation-aware request traces, and retain immutable resumable artifacts
• keep recommendations conservative: require unchanged correctness, a successful pair, complete request telemetry, material improvement, and no regression in the other measured dimensions
• preserve normal multi-version Harbor caches by selecting the newest requested task version
• fail closed when invalid host protocol overrides cross the Harbor host-cell boundary

Stack

• Runtime protocol support: feat(runtime): support Kimi OpenAI protocol #1462 (merged)
• Request-trace evidence: feat(runtime): validate provider request traces #1464 (merged)
• Benchmark harness: this PR, rebuilt as one commit on the current main

Verification

• npm ci
• npm run build:test
• npm run test:dist
• focused launcher and Headless benchmark suites
• npm run typecheck
• npm run lint
• npm run format:check
• git diff --check
• complete local diff review and independent Codex review: no findings
• GitHub CI: typecheck, test, and e2e all passed on 6a9a8b89

No paid/account-plan benchmark was run. The Anthropic-compatible default remains unchanged.

Refs #1269.

[Astro-Han]

I do not think this PR is ready to merge in its current shape. The concern is not the line count by itself: the PR combines three independently owned seams—runtime protocol support, request-trace evidence, and the benchmark harness—and the concrete failures found during review align with those boundaries.

I recommend converting this PR back to draft and splitting it before continuing review.

Blocking findings

Runtime protocol support

• Empty reasoning is not preserved. model-adapter.ts drops reasoning_content: "" / reasoning: "" because it only emits a thinking event when the string is non-empty. Preserved-thinking backends require the field to round-trip even when empty, so a valid tool-call response can make the next request fail.
• The detected OpenAI reasoning dialect has the wrong lifetime. createKimiOpenAiTransport() stores it in a model-instance closure, while AiSdkBackend.send() resolves a new model for every user turn. An endpoint that returns reasoning works for later steps in the first turn, then the next turn starts again with reasoning_content.
• The new product-loop test covers multiple provider steps inside one send(), but not two user turns or an empty reasoning field, so both lifecycle failures remain green.

Request-trace evidence

• requestPayloadWithoutProviderOptionsHash was added as a required field while the capture remains schema version 1. The new reader silently drops valid traces created before this PR; reading a legacy v1 capture now produces captures: [].
• The comparison validates capture/attempt consistency internally, but does not bind the trace’s run/session/turn identity to the task execution that returned the path. The new success fixture itself uses task-specific runtime refs but fixed run-1 / session-1 trace events and still passes.
• Malformed or torn attempt records are silently skipped. If another attempt still references the capture, validation passes and request count, failures, latency, and token usage can be understated.

Benchmark harness and recommendation

• The immutable/resume fingerprint does not cover the full environment inherited by Harbor. Variables such as MAKA_CONTEXT_*, continuation settings, and stream/command timeouts can change between resumes, mixing different runtime policies into one supposedly single-variable comparison.
• The smoke comparison runs only after every paid task, repetition, and arm has completed. A shared request-shape drift therefore consumes the full run before failing instead of acting as a preflight gate.
• The documented kimi-for-coding option is currently guaranteed to fail that smoke check: Anthropic and OpenAI captures expose different SDK-level maxOutputTokens values even though the final wire limit is the same.
• The recommendation gate can produce openai_candidate when both arms fail every task, provided there are enough paired failures to pass the non-inferiority calculation. It also treats any one-metric improvement as sufficient, including a 1 ms latency difference while another metric regresses without bound.
• A terminal event without a trace poisons the immutable run: resume retrieves the same event and throws again instead of producing an invalid/not-cleared report.
• The default Harbor task cache root fails when a task has multiple cached versions, which is a normal local-cache state. The documented dry-run is reproducibly blocked by duplicate task IDs.
• The documented dry-run and paid-run commands do not set a stable run ID, so they validate and execute different manifests by default.

CI is green, and the happy-path product-loop coverage is useful, but the current tests do not establish the core experiment invariants: cross-turn reasoning replay, empty reasoning, legacy trace compatibility, execution identity, resume under environment changes, real cross-protocol captures, or complete ordered tool-call arguments/results.

Suggested split

1. Runtime protocol support

   • Existing Kimi provider protocol selection
   • Model/runtime/factory and OpenAI transport changes
   • Cross-turn reasoning, empty reasoning, tool pairing, and usage contract tests

2. Request-trace evidence

   • Protocol-independent request payload hash
   • Schema/version compatibility
   • Trace identity and fail-closed completeness validation
   • Reader and artifact compatibility tests

3. Benchmark harness

   • Kimi protocol A/B runner and Harbor wiring
   • Immutable manifest/resume behavior
   • Smoke preflight and recommendation policy
   • Launcher tests and documentation

The first two seams can be reviewed as independent flat PRs from main. The benchmark PR can then be stacked on them because it genuinely depends on both. Tests should remain with the behavior they verify rather than being split into separate test-only PRs.

This split is about ownership, verification, and independent rollback—not splitting for size alone. It should also make the remaining fixes substantially easier to reason about and review.

[Nyvo-io]

Rebuilt this PR according to the requested split. The runtime protocol layer is in merged #1462, the request-trace layer is in #1464, and this branch now contains the benchmark harness stacked on #1464 (c1917e8f) rather than the previous combined implementation.

The benchmark findings from the original review are covered: inherited runtime policy is fingerprinted, the paired smoke check runs before the remaining matrix, equivalent output limits are normalized, recommendations require successful comparable evidence and material non-regressing improvement, terminal missing-trace outcomes resume as stable invalid results, multi-version caches select the newest requested task, and dry/paid runs share an explicit immutable run ID. The host-cell protocol boundary also now preserves invalid values so Kimi validation fails closed.

Local full headless: 1394 tests, 1393 passed, 1 Pier-Python skip, 0 failed. Typecheck/lint/format/diff checks pass, the independent Codex review reported no findings, and GitHub typecheck/test/e2e are all green on 8ed7c034.

I am leaving #1451 as draft while #1464 remains its open dependency.

[Nyvo-io]

Ready for re-review at 6a9a8b8. The branch is now one benchmark-only commit on current main after #1462 and #1464 merged. The original harness findings were rechecked against the final diff; local build, full serial workspace tests, typecheck, lint, format, focused benchmark tests, and an independent Codex review all passed with no findings. GitHub typecheck, test, and e2e are green. No paid benchmark was run, and the Anthropic-compatible default is unchanged.

[Astro-Han]

Reviewed 6a9a8b89. The benchmark-only split looks right, but one experiment-validity issue remains:

• [P1] compareKimiProtocolSmokeTrace() checks only the first step === 0 capture and always reports requestCount: 1. Continuation traces can contain several provider requests, so a later request can change its shared history or non-protocol parameters without failing the smoke gate. That can produce an openai_candidate result from a confounded run. Please compare every capture in order, require matching request/step counts, and add a test where a request after the first one differs.

The earlier split, resume, and recommendation-gate concerns otherwise look addressed.

[Nyvo-io]

Ready for re-review at 730c6302. The smoke gate now compares every capture in original order, requires matching request counts and step sequences, checks shared/model/non-protocol evidence per request, and verifies both arms use the same provider connection while differing in provider_options. Continuation regressions cover later-history drift plus count and step mismatches. Local repository build, full serial workspace tests, typecheck, lint, format, diff checks, and independent Codex review all passed; GitHub typecheck, test, and e2e are green. No paid benchmark was run.

[Astro-Han]

Rechecked 730c6302. The earlier P1 is fixed: the smoke gate now compares every provider request in order and fails on count, step, shared-history, model, connection, non-protocol parameter, or provider-options drift. The continuation regressions cover the previously missed later-request case. Focused tests pass 26/26, CI is green, and I found no remaining P0–P3 issues.