feat(headless): Config.systemPrompt + runtime_error failure class by Astro-Han · Pull Request #62 · apache/maka

Astro-Han · 2026-06-19T08:36:04Z

Summary

Two fixes that unblock real-model benchmark smoke (Terminal-Bench on DeepSeek), exposed by the diagnosis in the prior session:

Config.systemPrompt — real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the 8192 output token limit, and never produce the required artifact. Adds systemPrompt?: string to Config; the registerBackends factory closure reads it and passes it to AiSdkBackend directly (same pattern as the desktop interactive path). Not routed through BackendFactoryContext.systemPrompt (that's the child-agent instruction channel) and not persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task.
failureClass: runtime_error — a backend that ends with complete(stopReason='error') without a preceding error event left the run ledger's failureClass as unknown, so benchmark scoring could not distinguish runtime failures from max_tokens / incomplete_tool_calls. Classifies it as runtime_error instead.

Both reviewed by Codex and Claude (consult mode) — confirmed headless-only / runtime-pass / no-persistence is the right scope.

Test plan

@maka/headless 91/91 (2 new tests: factory closure reads config.systemPrompt; undefined when omitted)
@maka/runtime 586/586 (1 new test: complete(stopReason=error) → runtime_error not unknown)
npm run typecheck clean

Real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the output token limit, and never produce the required artifact. Add systemPrompt?: string to Config. The registerBackends factory closure reads it and passes it to AiSdkBackend directly — same pattern as the desktop interactive path. systemPrompt is NOT routed through BackendFactoryContext.systemPrompt (that channel is the child-agent instruction, not the main-session prompt) and is NOT persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task. Reviewed by Codex and Claude (consult mode) — both confirmed headless-only / runtime-pass / no-persistence is the right scope.

…t unknown When a backend ends with stopReason='error' but never emits a preceding error event (observed with DeepSeek-reasoner after it tried to call an unavailable tool and entered a reasoning loop), the run ledger's failureClass fell back to 'unknown' — making benchmark failures unattributable. Now turnStatusFromEvent returns errorClass 'runtime_error' for stopReason='error', and recordSessionEvent calls markRunFailed so finalize does not fall back to 'unknown'. This lets the benchmark scorer distinguish runtime failures from max_tokens / verification_failed.

…mment Codex [P2]: BENCHMARK_BASE_SYSTEM_PROMPT said 'You do not have a Bash tool' but the standard isolated headless surface (buildIsolatedHeadlessTools) includes an isolated Bash. Reworded to 'do not self-test or run verification scripts' — acknowledges Bash exists for producing artifacts, steers away from self-grading. Claude [P3]: complete(stopReason=error) path now emits two run_failed events (markRunFailed + finishRun), same as the error-event path already did. Added a comment making the conscious choice explicit.

Claude review follow-up: the existing tests proved the factory closure can READ config.systemPrompt, but not that it actually PASSES it to the backend constructor's systemPrompt parameter — the exact seam a real AiSdkBackend factory uses. Adds an ai-sdk stub backend that records its constructor systemPrompt input, and a test that runs runExperiment with registerBackends wiring config.systemPrompt → backend ctor (mirroring the desktop pattern). Verifies the prompt string arrives at the backend constructor intact.

…ror (#63) Fresh-eye review (Codex P1 / Claude P3) found that PR #62 only fixed the AgentRunStore ledger failureClass, not the InvocationResult.failure.class that benchmark ResultRecord reads. A complete(stopReason=error) with no preceding error event still surfaced as errorClass='failed' in ResultRecord — indistinguishable from other failures. Root cause: failureFromTerminalEvent returned class=status (i.e. 'failed') for any failed terminal, ignoring whether error content carried a precise reason. The error-content branch (which extracts content.reason) was unreachable for terminal events because the terminal check ran first. Fix: in failureFromTerminalEvent, when status='failed': - with error content reason/code → use that (e.g. 'tool_failed') - without error content → 'runtime_error' (not 'failed') This makes the invocation path consistent with the run ledger, so ResultRecord.errorClass reads 'runtime_error' for the bare-error scenario. Tests: - runtime: updated locked test (failed+error content w/o reason → runtime_error), added reason-code test, added bare-failed-terminal test - headless: added end-to-end test verifying ResultRecord.errorClass='runtime_error' for a backend that emits only complete(stopReason=error)

Astro-Han · 2026-06-19T08:36:04Z

Summary

Two fixes that unblock real-model benchmark smoke (Terminal-Bench on DeepSeek), exposed by the diagnosis in the prior session:

Config.systemPrompt — real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the 8192 output token limit, and never produce the required artifact. Adds systemPrompt?: string to Config; the registerBackends factory closure reads it and passes it to AiSdkBackend directly (same pattern as the desktop interactive path). Not routed through BackendFactoryContext.systemPrompt (that's the child-agent instruction channel) and not persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task.
failureClass: runtime_error — a backend that ends with complete(stopReason='error') without a preceding error event left the run ledger's failureClass as unknown, so benchmark scoring could not distinguish runtime failures from max_tokens / incomplete_tool_calls. Classifies it as runtime_error instead.

Both reviewed by Codex and Claude (consult mode) — confirmed headless-only / runtime-pass / no-persistence is the right scope.

Test plan

@maka/headless 91/91 (2 new tests: factory closure reads config.systemPrompt; undefined when omitted)
@maka/runtime 586/586 (1 new test: complete(stopReason=error) → runtime_error not unknown)
npm run typecheck clean

Real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the output token limit, and never produce the required artifact. Add systemPrompt?: string to Config. The registerBackends factory closure reads it and passes it to AiSdkBackend directly — same pattern as the desktop interactive path. systemPrompt is NOT routed through BackendFactoryContext.systemPrompt (that channel is the child-agent instruction, not the main-session prompt) and is NOT persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task. Reviewed by Codex and Claude (consult mode) — both confirmed headless-only / runtime-pass / no-persistence is the right scope.

…t unknown When a backend ends with stopReason='error' but never emits a preceding error event (observed with DeepSeek-reasoner after it tried to call an unavailable tool and entered a reasoning loop), the run ledger's failureClass fell back to 'unknown' — making benchmark failures unattributable. Now turnStatusFromEvent returns errorClass 'runtime_error' for stopReason='error', and recordSessionEvent calls markRunFailed so finalize does not fall back to 'unknown'. This lets the benchmark scorer distinguish runtime failures from max_tokens / verification_failed.

…mment Codex [P2]: BENCHMARK_BASE_SYSTEM_PROMPT said 'You do not have a Bash tool' but the standard isolated headless surface (buildIsolatedHeadlessTools) includes an isolated Bash. Reworded to 'do not self-test or run verification scripts' — acknowledges Bash exists for producing artifacts, steers away from self-grading. Claude [P3]: complete(stopReason=error) path now emits two run_failed events (markRunFailed + finishRun), same as the error-event path already did. Added a comment making the conscious choice explicit.

Claude review follow-up: the existing tests proved the factory closure can READ config.systemPrompt, but not that it actually PASSES it to the backend constructor's systemPrompt parameter — the exact seam a real AiSdkBackend factory uses. Adds an ai-sdk stub backend that records its constructor systemPrompt input, and a test that runs runExperiment with registerBackends wiring config.systemPrompt → backend ctor (mirroring the desktop pattern). Verifies the prompt string arrives at the backend constructor intact.

…ror (#63) Fresh-eye review (Codex P1 / Claude P3) found that PR #62 only fixed the AgentRunStore ledger failureClass, not the InvocationResult.failure.class that benchmark ResultRecord reads. A complete(stopReason=error) with no preceding error event still surfaced as errorClass='failed' in ResultRecord — indistinguishable from other failures. Root cause: failureFromTerminalEvent returned class=status (i.e. 'failed') for any failed terminal, ignoring whether error content carried a precise reason. The error-content branch (which extracts content.reason) was unreachable for terminal events because the terminal check ran first. Fix: in failureFromTerminalEvent, when status='failed': - with error content reason/code → use that (e.g. 'tool_failed') - without error content → 'runtime_error' (not 'failed') This makes the invocation path consistent with the run ledger, so ResultRecord.errorClass reads 'runtime_error' for the bare-error scenario. Tests: - runtime: updated locked test (failed+error content w/o reason → runtime_error), added reason-code test, added bare-failed-terminal test - headless: added end-to-end test verifying ResultRecord.errorClass='runtime_error' for a backend that emits only complete(stopReason=error)

Astro-Han · 2026-06-19T08:36:04Z

Summary

Two fixes that unblock real-model benchmark smoke (Terminal-Bench on DeepSeek), exposed by the diagnosis in the prior session:

Config.systemPrompt — real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the 8192 output token limit, and never produce the required artifact. Adds systemPrompt?: string to Config; the registerBackends factory closure reads it and passes it to AiSdkBackend directly (same pattern as the desktop interactive path). Not routed through BackendFactoryContext.systemPrompt (that's the child-agent instruction channel) and not persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task.
failureClass: runtime_error — a backend that ends with complete(stopReason='error') without a preceding error event left the run ledger's failureClass as unknown, so benchmark scoring could not distinguish runtime failures from max_tokens / incomplete_tool_calls. Classifies it as runtime_error instead.

Both reviewed by Codex and Claude (consult mode) — confirmed headless-only / runtime-pass / no-persistence is the right scope.

Test plan

@maka/headless 91/91 (2 new tests: factory closure reads config.systemPrompt; undefined when omitted)
@maka/runtime 586/586 (1 new test: complete(stopReason=error) → runtime_error not unknown)
npm run typecheck clean

Real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the output token limit, and never produce the required artifact. Add systemPrompt?: string to Config. The registerBackends factory closure reads it and passes it to AiSdkBackend directly — same pattern as the desktop interactive path. systemPrompt is NOT routed through BackendFactoryContext.systemPrompt (that channel is the child-agent instruction, not the main-session prompt) and is NOT persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task. Reviewed by Codex and Claude (consult mode) — both confirmed headless-only / runtime-pass / no-persistence is the right scope.

…t unknown When a backend ends with stopReason='error' but never emits a preceding error event (observed with DeepSeek-reasoner after it tried to call an unavailable tool and entered a reasoning loop), the run ledger's failureClass fell back to 'unknown' — making benchmark failures unattributable. Now turnStatusFromEvent returns errorClass 'runtime_error' for stopReason='error', and recordSessionEvent calls markRunFailed so finalize does not fall back to 'unknown'. This lets the benchmark scorer distinguish runtime failures from max_tokens / verification_failed.

…mment Codex [P2]: BENCHMARK_BASE_SYSTEM_PROMPT said 'You do not have a Bash tool' but the standard isolated headless surface (buildIsolatedHeadlessTools) includes an isolated Bash. Reworded to 'do not self-test or run verification scripts' — acknowledges Bash exists for producing artifacts, steers away from self-grading. Claude [P3]: complete(stopReason=error) path now emits two run_failed events (markRunFailed + finishRun), same as the error-event path already did. Added a comment making the conscious choice explicit.

Claude review follow-up: the existing tests proved the factory closure can READ config.systemPrompt, but not that it actually PASSES it to the backend constructor's systemPrompt parameter — the exact seam a real AiSdkBackend factory uses. Adds an ai-sdk stub backend that records its constructor systemPrompt input, and a test that runs runExperiment with registerBackends wiring config.systemPrompt → backend ctor (mirroring the desktop pattern). Verifies the prompt string arrives at the backend constructor intact.

…ror (#63) Fresh-eye review (Codex P1 / Claude P3) found that PR #62 only fixed the AgentRunStore ledger failureClass, not the InvocationResult.failure.class that benchmark ResultRecord reads. A complete(stopReason=error) with no preceding error event still surfaced as errorClass='failed' in ResultRecord — indistinguishable from other failures. Root cause: failureFromTerminalEvent returned class=status (i.e. 'failed') for any failed terminal, ignoring whether error content carried a precise reason. The error-content branch (which extracts content.reason) was unreachable for terminal events because the terminal check ran first. Fix: in failureFromTerminalEvent, when status='failed': - with error content reason/code → use that (e.g. 'tool_failed') - without error content → 'runtime_error' (not 'failed') This makes the invocation path consistent with the run ledger, so ResultRecord.errorClass reads 'runtime_error' for the bare-error scenario. Tests: - runtime: updated locked test (failed+error content w/o reason → runtime_error), added reason-code test, added bare-failed-terminal test - headless: added end-to-end test verifying ResultRecord.errorClass='runtime_error' for a backend that emits only complete(stopReason=error)

Astro-Han · 2026-06-19T08:36:04Z

Summary

Two fixes that unblock real-model benchmark smoke (Terminal-Bench on DeepSeek), exposed by the diagnosis in the prior session:

Config.systemPrompt — real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the 8192 output token limit, and never produce the required artifact. Adds systemPrompt?: string to Config; the registerBackends factory closure reads it and passes it to AiSdkBackend directly (same pattern as the desktop interactive path). Not routed through BackendFactoryContext.systemPrompt (that's the child-agent instruction channel) and not persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task.
failureClass: runtime_error — a backend that ends with complete(stopReason='error') without a preceding error event left the run ledger's failureClass as unknown, so benchmark scoring could not distinguish runtime failures from max_tokens / incomplete_tool_calls. Classifies it as runtime_error instead.

Both reviewed by Codex and Claude (consult mode) — confirmed headless-only / runtime-pass / no-persistence is the right scope.

Test plan

@maka/headless 91/91 (2 new tests: factory closure reads config.systemPrompt; undefined when omitted)
@maka/runtime 586/586 (1 new test: complete(stopReason=error) → runtime_error not unknown)
npm run typecheck clean

Real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the output token limit, and never produce the required artifact. Add systemPrompt?: string to Config. The registerBackends factory closure reads it and passes it to AiSdkBackend directly — same pattern as the desktop interactive path. systemPrompt is NOT routed through BackendFactoryContext.systemPrompt (that channel is the child-agent instruction, not the main-session prompt) and is NOT persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task. Reviewed by Codex and Claude (consult mode) — both confirmed headless-only / runtime-pass / no-persistence is the right scope.

…t unknown When a backend ends with stopReason='error' but never emits a preceding error event (observed with DeepSeek-reasoner after it tried to call an unavailable tool and entered a reasoning loop), the run ledger's failureClass fell back to 'unknown' — making benchmark failures unattributable. Now turnStatusFromEvent returns errorClass 'runtime_error' for stopReason='error', and recordSessionEvent calls markRunFailed so finalize does not fall back to 'unknown'. This lets the benchmark scorer distinguish runtime failures from max_tokens / verification_failed.

…mment Codex [P2]: BENCHMARK_BASE_SYSTEM_PROMPT said 'You do not have a Bash tool' but the standard isolated headless surface (buildIsolatedHeadlessTools) includes an isolated Bash. Reworded to 'do not self-test or run verification scripts' — acknowledges Bash exists for producing artifacts, steers away from self-grading. Claude [P3]: complete(stopReason=error) path now emits two run_failed events (markRunFailed + finishRun), same as the error-event path already did. Added a comment making the conscious choice explicit.

Claude review follow-up: the existing tests proved the factory closure can READ config.systemPrompt, but not that it actually PASSES it to the backend constructor's systemPrompt parameter — the exact seam a real AiSdkBackend factory uses. Adds an ai-sdk stub backend that records its constructor systemPrompt input, and a test that runs runExperiment with registerBackends wiring config.systemPrompt → backend ctor (mirroring the desktop pattern). Verifies the prompt string arrives at the backend constructor intact.

…ror (#63) Fresh-eye review (Codex P1 / Claude P3) found that PR #62 only fixed the AgentRunStore ledger failureClass, not the InvocationResult.failure.class that benchmark ResultRecord reads. A complete(stopReason=error) with no preceding error event still surfaced as errorClass='failed' in ResultRecord — indistinguishable from other failures. Root cause: failureFromTerminalEvent returned class=status (i.e. 'failed') for any failed terminal, ignoring whether error content carried a precise reason. The error-content branch (which extracts content.reason) was unreachable for terminal events because the terminal check ran first. Fix: in failureFromTerminalEvent, when status='failed': - with error content reason/code → use that (e.g. 'tool_failed') - without error content → 'runtime_error' (not 'failed') This makes the invocation path consistent with the run ledger, so ResultRecord.errorClass reads 'runtime_error' for the bare-error scenario. Tests: - runtime: updated locked test (failed+error content w/o reason → runtime_error), added reason-code test, added bare-failed-terminal test - headless: added end-to-end test verifying ResultRecord.errorClass='runtime_error' for a backend that emits only complete(stopReason=error)

Astro-Han · 2026-06-19T08:36:04Z

Summary

Two fixes that unblock real-model benchmark smoke (Terminal-Bench on DeepSeek), exposed by the diagnosis in the prior session:

Config.systemPrompt — real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the 8192 output token limit, and never produce the required artifact. Adds systemPrompt?: string to Config; the registerBackends factory closure reads it and passes it to AiSdkBackend directly (same pattern as the desktop interactive path). Not routed through BackendFactoryContext.systemPrompt (that's the child-agent instruction channel) and not persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task.
failureClass: runtime_error — a backend that ends with complete(stopReason='error') without a preceding error event left the run ledger's failureClass as unknown, so benchmark scoring could not distinguish runtime failures from max_tokens / incomplete_tool_calls. Classifies it as runtime_error instead.

Both reviewed by Codex and Claude (consult mode) — confirmed headless-only / runtime-pass / no-persistence is the right scope.

Test plan

@maka/headless 91/91 (2 new tests: factory closure reads config.systemPrompt; undefined when omitted)
@maka/runtime 586/586 (1 new test: complete(stopReason=error) → runtime_error not unknown)
npm run typecheck clean

Real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the output token limit, and never produce the required artifact. Add systemPrompt?: string to Config. The registerBackends factory closure reads it and passes it to AiSdkBackend directly — same pattern as the desktop interactive path. systemPrompt is NOT routed through BackendFactoryContext.systemPrompt (that channel is the child-agent instruction, not the main-session prompt) and is NOT persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task. Reviewed by Codex and Claude (consult mode) — both confirmed headless-only / runtime-pass / no-persistence is the right scope.

…t unknown When a backend ends with stopReason='error' but never emits a preceding error event (observed with DeepSeek-reasoner after it tried to call an unavailable tool and entered a reasoning loop), the run ledger's failureClass fell back to 'unknown' — making benchmark failures unattributable. Now turnStatusFromEvent returns errorClass 'runtime_error' for stopReason='error', and recordSessionEvent calls markRunFailed so finalize does not fall back to 'unknown'. This lets the benchmark scorer distinguish runtime failures from max_tokens / verification_failed.

…mment Codex [P2]: BENCHMARK_BASE_SYSTEM_PROMPT said 'You do not have a Bash tool' but the standard isolated headless surface (buildIsolatedHeadlessTools) includes an isolated Bash. Reworded to 'do not self-test or run verification scripts' — acknowledges Bash exists for producing artifacts, steers away from self-grading. Claude [P3]: complete(stopReason=error) path now emits two run_failed events (markRunFailed + finishRun), same as the error-event path already did. Added a comment making the conscious choice explicit.

Claude review follow-up: the existing tests proved the factory closure can READ config.systemPrompt, but not that it actually PASSES it to the backend constructor's systemPrompt parameter — the exact seam a real AiSdkBackend factory uses. Adds an ai-sdk stub backend that records its constructor systemPrompt input, and a test that runs runExperiment with registerBackends wiring config.systemPrompt → backend ctor (mirroring the desktop pattern). Verifies the prompt string arrives at the backend constructor intact.

…ror (#63) Fresh-eye review (Codex P1 / Claude P3) found that PR #62 only fixed the AgentRunStore ledger failureClass, not the InvocationResult.failure.class that benchmark ResultRecord reads. A complete(stopReason=error) with no preceding error event still surfaced as errorClass='failed' in ResultRecord — indistinguishable from other failures. Root cause: failureFromTerminalEvent returned class=status (i.e. 'failed') for any failed terminal, ignoring whether error content carried a precise reason. The error-content branch (which extracts content.reason) was unreachable for terminal events because the terminal check ran first. Fix: in failureFromTerminalEvent, when status='failed': - with error content reason/code → use that (e.g. 'tool_failed') - without error content → 'runtime_error' (not 'failed') This makes the invocation path consistent with the run ledger, so ResultRecord.errorClass reads 'runtime_error' for the bare-error scenario. Tests: - runtime: updated locked test (failed+error content w/o reason → runtime_error), added reason-code test, added bare-failed-terminal test - headless: added end-to-end test verifying ResultRecord.errorClass='runtime_error' for a backend that emits only complete(stopReason=error)

Astro-Han · 2026-06-19T08:36:04Z

Summary

Two fixes that unblock real-model benchmark smoke (Terminal-Bench on DeepSeek), exposed by the diagnosis in the prior session:

Config.systemPrompt — real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the 8192 output token limit, and never produce the required artifact. Adds systemPrompt?: string to Config; the registerBackends factory closure reads it and passes it to AiSdkBackend directly (same pattern as the desktop interactive path). Not routed through BackendFactoryContext.systemPrompt (that's the child-agent instruction channel) and not persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task.
failureClass: runtime_error — a backend that ends with complete(stopReason='error') without a preceding error event left the run ledger's failureClass as unknown, so benchmark scoring could not distinguish runtime failures from max_tokens / incomplete_tool_calls. Classifies it as runtime_error instead.

Both reviewed by Codex and Claude (consult mode) — confirmed headless-only / runtime-pass / no-persistence is the right scope.

Test plan

@maka/headless 91/91 (2 new tests: factory closure reads config.systemPrompt; undefined when omitted)
@maka/runtime 586/586 (1 new test: complete(stopReason=error) → runtime_error not unknown)
npm run typecheck clean

Real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the output token limit, and never produce the required artifact. Add systemPrompt?: string to Config. The registerBackends factory closure reads it and passes it to AiSdkBackend directly — same pattern as the desktop interactive path. systemPrompt is NOT routed through BackendFactoryContext.systemPrompt (that channel is the child-agent instruction, not the main-session prompt) and is NOT persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task. Reviewed by Codex and Claude (consult mode) — both confirmed headless-only / runtime-pass / no-persistence is the right scope.

…t unknown When a backend ends with stopReason='error' but never emits a preceding error event (observed with DeepSeek-reasoner after it tried to call an unavailable tool and entered a reasoning loop), the run ledger's failureClass fell back to 'unknown' — making benchmark failures unattributable. Now turnStatusFromEvent returns errorClass 'runtime_error' for stopReason='error', and recordSessionEvent calls markRunFailed so finalize does not fall back to 'unknown'. This lets the benchmark scorer distinguish runtime failures from max_tokens / verification_failed.

…mment Codex [P2]: BENCHMARK_BASE_SYSTEM_PROMPT said 'You do not have a Bash tool' but the standard isolated headless surface (buildIsolatedHeadlessTools) includes an isolated Bash. Reworded to 'do not self-test or run verification scripts' — acknowledges Bash exists for producing artifacts, steers away from self-grading. Claude [P3]: complete(stopReason=error) path now emits two run_failed events (markRunFailed + finishRun), same as the error-event path already did. Added a comment making the conscious choice explicit.

Claude review follow-up: the existing tests proved the factory closure can READ config.systemPrompt, but not that it actually PASSES it to the backend constructor's systemPrompt parameter — the exact seam a real AiSdkBackend factory uses. Adds an ai-sdk stub backend that records its constructor systemPrompt input, and a test that runs runExperiment with registerBackends wiring config.systemPrompt → backend ctor (mirroring the desktop pattern). Verifies the prompt string arrives at the backend constructor intact.

…ror (#63) Fresh-eye review (Codex P1 / Claude P3) found that PR #62 only fixed the AgentRunStore ledger failureClass, not the InvocationResult.failure.class that benchmark ResultRecord reads. A complete(stopReason=error) with no preceding error event still surfaced as errorClass='failed' in ResultRecord — indistinguishable from other failures. Root cause: failureFromTerminalEvent returned class=status (i.e. 'failed') for any failed terminal, ignoring whether error content carried a precise reason. The error-content branch (which extracts content.reason) was unreachable for terminal events because the terminal check ran first. Fix: in failureFromTerminalEvent, when status='failed': - with error content reason/code → use that (e.g. 'tool_failed') - without error content → 'runtime_error' (not 'failed') This makes the invocation path consistent with the run ledger, so ResultRecord.errorClass reads 'runtime_error' for the bare-error scenario. Tests: - runtime: updated locked test (failed+error content w/o reason → runtime_error), added reason-code test, added bare-failed-terminal test - headless: added end-to-end test verifying ResultRecord.errorClass='runtime_error' for a backend that emits only complete(stopReason=error)

Astro-Han · 2026-06-19T08:36:04Z

Summary

Two fixes that unblock real-model benchmark smoke (Terminal-Bench on DeepSeek), exposed by the diagnosis in the prior session:

Config.systemPrompt — real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the 8192 output token limit, and never produce the required artifact. Adds systemPrompt?: string to Config; the registerBackends factory closure reads it and passes it to AiSdkBackend directly (same pattern as the desktop interactive path). Not routed through BackendFactoryContext.systemPrompt (that's the child-agent instruction channel) and not persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task.
failureClass: runtime_error — a backend that ends with complete(stopReason='error') without a preceding error event left the run ledger's failureClass as unknown, so benchmark scoring could not distinguish runtime failures from max_tokens / incomplete_tool_calls. Classifies it as runtime_error instead.

Both reviewed by Codex and Claude (consult mode) — confirmed headless-only / runtime-pass / no-persistence is the right scope.

Test plan

@maka/headless 91/91 (2 new tests: factory closure reads config.systemPrompt; undefined when omitted)
@maka/runtime 586/586 (1 new test: complete(stopReason=error) → runtime_error not unknown)
npm run typecheck clean

Real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the output token limit, and never produce the required artifact. Add systemPrompt?: string to Config. The registerBackends factory closure reads it and passes it to AiSdkBackend directly — same pattern as the desktop interactive path. systemPrompt is NOT routed through BackendFactoryContext.systemPrompt (that channel is the child-agent instruction, not the main-session prompt) and is NOT persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task. Reviewed by Codex and Claude (consult mode) — both confirmed headless-only / runtime-pass / no-persistence is the right scope.

…t unknown When a backend ends with stopReason='error' but never emits a preceding error event (observed with DeepSeek-reasoner after it tried to call an unavailable tool and entered a reasoning loop), the run ledger's failureClass fell back to 'unknown' — making benchmark failures unattributable. Now turnStatusFromEvent returns errorClass 'runtime_error' for stopReason='error', and recordSessionEvent calls markRunFailed so finalize does not fall back to 'unknown'. This lets the benchmark scorer distinguish runtime failures from max_tokens / verification_failed.

…mment Codex [P2]: BENCHMARK_BASE_SYSTEM_PROMPT said 'You do not have a Bash tool' but the standard isolated headless surface (buildIsolatedHeadlessTools) includes an isolated Bash. Reworded to 'do not self-test or run verification scripts' — acknowledges Bash exists for producing artifacts, steers away from self-grading. Claude [P3]: complete(stopReason=error) path now emits two run_failed events (markRunFailed + finishRun), same as the error-event path already did. Added a comment making the conscious choice explicit.

Claude review follow-up: the existing tests proved the factory closure can READ config.systemPrompt, but not that it actually PASSES it to the backend constructor's systemPrompt parameter — the exact seam a real AiSdkBackend factory uses. Adds an ai-sdk stub backend that records its constructor systemPrompt input, and a test that runs runExperiment with registerBackends wiring config.systemPrompt → backend ctor (mirroring the desktop pattern). Verifies the prompt string arrives at the backend constructor intact.

…ror (#63) Fresh-eye review (Codex P1 / Claude P3) found that PR #62 only fixed the AgentRunStore ledger failureClass, not the InvocationResult.failure.class that benchmark ResultRecord reads. A complete(stopReason=error) with no preceding error event still surfaced as errorClass='failed' in ResultRecord — indistinguishable from other failures. Root cause: failureFromTerminalEvent returned class=status (i.e. 'failed') for any failed terminal, ignoring whether error content carried a precise reason. The error-content branch (which extracts content.reason) was unreachable for terminal events because the terminal check ran first. Fix: in failureFromTerminalEvent, when status='failed': - with error content reason/code → use that (e.g. 'tool_failed') - without error content → 'runtime_error' (not 'failed') This makes the invocation path consistent with the run ledger, so ResultRecord.errorClass reads 'runtime_error' for the bare-error scenario. Tests: - runtime: updated locked test (failed+error content w/o reason → runtime_error), added reason-code test, added bare-failed-terminal test - headless: added end-to-end test verifying ResultRecord.errorClass='runtime_error' for a backend that emits only complete(stopReason=error)

Astro-Han · 2026-06-19T08:36:04Z

Summary

Two fixes that unblock real-model benchmark smoke (Terminal-Bench on DeepSeek), exposed by the diagnosis in the prior session:

Config.systemPrompt — real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the 8192 output token limit, and never produce the required artifact. Adds systemPrompt?: string to Config; the registerBackends factory closure reads it and passes it to AiSdkBackend directly (same pattern as the desktop interactive path). Not routed through BackendFactoryContext.systemPrompt (that's the child-agent instruction channel) and not persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task.
failureClass: runtime_error — a backend that ends with complete(stopReason='error') without a preceding error event left the run ledger's failureClass as unknown, so benchmark scoring could not distinguish runtime failures from max_tokens / incomplete_tool_calls. Classifies it as runtime_error instead.

Both reviewed by Codex and Claude (consult mode) — confirmed headless-only / runtime-pass / no-persistence is the right scope.

Test plan

@maka/headless 91/91 (2 new tests: factory closure reads config.systemPrompt; undefined when omitted)
@maka/runtime 586/586 (1 new test: complete(stopReason=error) → runtime_error not unknown)
npm run typecheck clean

Real models running benchmark tasks without a system prompt narrate their reasoning in text instead of calling tools, hit the output token limit, and never produce the required artifact. Add systemPrompt?: string to Config. The registerBackends factory closure reads it and passes it to AiSdkBackend directly — same pattern as the desktop interactive path. systemPrompt is NOT routed through BackendFactoryContext.systemPrompt (that channel is the child-agent instruction, not the main-session prompt) and is NOT persisted into SessionHeader (benchmark sessions are throwaway). Also exports BENCHMARK_BASE_SYSTEM_PROMPT, a generic tool-first prefix verified against DeepSeek-chat on a Terminal-Bench regex task. Reviewed by Codex and Claude (consult mode) — both confirmed headless-only / runtime-pass / no-persistence is the right scope.

…t unknown When a backend ends with stopReason='error' but never emits a preceding error event (observed with DeepSeek-reasoner after it tried to call an unavailable tool and entered a reasoning loop), the run ledger's failureClass fell back to 'unknown' — making benchmark failures unattributable. Now turnStatusFromEvent returns errorClass 'runtime_error' for stopReason='error', and recordSessionEvent calls markRunFailed so finalize does not fall back to 'unknown'. This lets the benchmark scorer distinguish runtime failures from max_tokens / verification_failed.

…mment Codex [P2]: BENCHMARK_BASE_SYSTEM_PROMPT said 'You do not have a Bash tool' but the standard isolated headless surface (buildIsolatedHeadlessTools) includes an isolated Bash. Reworded to 'do not self-test or run verification scripts' — acknowledges Bash exists for producing artifacts, steers away from self-grading. Claude [P3]: complete(stopReason=error) path now emits two run_failed events (markRunFailed + finishRun), same as the error-event path already did. Added a comment making the conscious choice explicit.

Claude review follow-up: the existing tests proved the factory closure can READ config.systemPrompt, but not that it actually PASSES it to the backend constructor's systemPrompt parameter — the exact seam a real AiSdkBackend factory uses. Adds an ai-sdk stub backend that records its constructor systemPrompt input, and a test that runs runExperiment with registerBackends wiring config.systemPrompt → backend ctor (mirroring the desktop pattern). Verifies the prompt string arrives at the backend constructor intact.

…ror (#63) Fresh-eye review (Codex P1 / Claude P3) found that PR #62 only fixed the AgentRunStore ledger failureClass, not the InvocationResult.failure.class that benchmark ResultRecord reads. A complete(stopReason=error) with no preceding error event still surfaced as errorClass='failed' in ResultRecord — indistinguishable from other failures. Root cause: failureFromTerminalEvent returned class=status (i.e. 'failed') for any failed terminal, ignoring whether error content carried a precise reason. The error-content branch (which extracts content.reason) was unreachable for terminal events because the terminal check ran first. Fix: in failureFromTerminalEvent, when status='failed': - with error content reason/code → use that (e.g. 'tool_failed') - without error content → 'runtime_error' (not 'failed') This makes the invocation path consistent with the run ledger, so ResultRecord.errorClass reads 'runtime_error' for the bare-error scenario. Tests: - runtime: updated locked test (failed+error content w/o reason → runtime_error), added reason-code test, added bare-failed-terminal test - headless: added end-to-end test verifying ResultRecord.errorClass='runtime_error' for a backend that emits only complete(stopReason=error)

Astro-Han added 4 commits June 19, 2026 16:35

Astro-Han merged commit ab61bc5 into mainJun 19, 2026