docs: Terminal-Bench 2.1 Maka vs Kimi Code v11 results

[Astro-Han]

Summary

Docs-only: English write-up of Terminal-Bench 2.1 harness A/B run k3-maka-vs-kimi-code-tbench-2.1-full-v11 (Maka vs Kimi Code, both k3 / max thinking), plus a redaction-minimal accepted-outcome CSV.

Full report:docs/eval/terminal-bench-2.1-maka-vs-kimi-code-v11.md

Results

	Maka	Kimi Code
Pass@1 (accepted)	62/89 (69.7%)	53/89 (59.6%)
Mechanical first verifier	62/89	50/89 (56.2%)
Deadline-killed	28/89	33/89
Pass rate, finished in time	95.1%	85.7%
Pass rate when killed	14.3%	15.2%
List-price cost (88 pairs)	$52.72	$38.40

Official reference: 88.3% for Kimi K3 + KimiCode harness (tech blog, accessed 2026-07-19).

Kimi Code's accepted 53/89 is operator-adjusted (mechanical first verifier = 50/89). Three score-changing recoveries: prove-plus-comm, hf-model-inference, pypi-server. Run admissions: 203 physical → 178 accepted (25 recovery). Single-repetition; descriptive, not statistical.

Absolute scores are throughput-limited

Decode was ~34–37 output tok/s with ~14 s median first-token latency against task-native agent budgets. About a third of tasks per arm hit the deadline mid-generation and rarely passed (14–15%).

Finished-in-time pass rates (95.1% / 85.7%; 95.9% / 89.8% on the 49 tasks both arms finished) sit around the official 88.3%. Under healthy throughput, Maka would plausibly reach or exceed that reference and Kimi Code land near it — subsets are self-selected, so a re-run is decisive.

Official frontier is already compressed (~88.8% / 88.3%). Unconstrained subsets pass at 85.7–95.9%. TB 2.1 is close to saturated at the frontier; harder benchmarks will carry more signal.

Relative gap is harness asymmetry

Thinking effort, thinking retention, and wall-clock are aligned (same degraded network path). The +10.1 pp (Maka) persists among finished tasks (95.1% vs 85.7%). Differences below are listed, not ranked as score causes.

#	Difference	Notes
1	Context-budget tool-result prune (Maka only)	2048-token threshold ON; Kimi Code has no equivalent policy. Live 89/89; diagnostics 21/57 completed: 576 rewrites, ~1.87M est. tokens not re-sent (lower bound; aborted cells unmetered). Score impact not estimated (no prune-off arm).
2	System prompt	Empty vs ~20KB product main-agent template
3	Tool surface	Narrow foreground Bash vs full product CLI
4	Completion contract	Task+schemas vs product “tell the user you’re done” norms

Background Bash is not the exclusive-win story (2/12 Maka-only tasks used run_in_background on Kimi). Heavy-task self-check is in the build but was off (zero tool/gate events). Recorded cost is $0 (account-plan); list-price uses official K3 API rates.

Verification

• Load-bearing numbers recomputed from the frozen run: controller/results.jsonl{,.attempts.jsonl}, trial result.json, Kimi provider-request-telemetry.json, Maka maka-cell-output.json / runtime streams, harness-ab-report.json, task task.toml budgets
• Official 88.3% cross-checked via secondary sources quoting the tech blog table
• CSV 89/89 match to accepted outcomes; SHA-256 integrity table in the doc
• External review (Codex + Claude Fable); P0–P2 findings resolved
• Docs-only; no code, tests, or raw eval artifacts; git diff --check clean