Uh oh!
There was an error while loading. Please reload this page.
docs: publish DeepSeek Flash DeepSWE 1.1 comparison - #1848
Conversation
English write-up of harness A/B run deepseek-v4-flash-maka-vs-opencode-deepswe-full-v1 (Maka vs OpenCode, both on DeepSeek V4 Flash at max reasoning effort), plus the committed per-task outcome and failure-class CSV. Headline: Maka 52/111 (46.85%) vs OpenCode 40/112 (35.71%). On the 110 fully paired tasks the delta is 12.73 pp favoring Maka, but the exact two-sided McNemar test over the 54 discordant pairs gives p = 0.0759 -- it does not reach the 0.05 threshold, so the pass-rate difference is not statistically distinguishable on this suite. Cost per pass does differ materially: $0.2358 vs $0.3196 (OpenCode 35.5% higher), driven by effectiveness rather than resource efficiency. This contrasts with the Terminal-Bench 2.1 run (16:4 discordant, p = 0.0118) at comparable sample size, supporting the conclusion that harness advantage is task-distribution dependent.
Trajectory-level analysis of maka's 59 failures on the DeepSWE 1.1 full run, from 20,353 tool calls, 18,439 provider steps and 47.5M chars of final-step prompts. Three findings that change where effort should go: 1. Secret redaction is corrupting source fed to the model. 2,055 [redacted] spans across 113 prompts; only 20 (0.9%) are real credentials. The rest hit lexer 'Token:', hash-map 'Key:' and git SHAs. Three one-line rule fixes remove ~92% of the false positives. 2. Failures have no behavioural separator from passes -- steps, test invocations, edits and re-reads are statistically identical, and failures spend MORE reasoning. All 59 end self-declared complete, zero give-ups. This is a spec-comprehension gap, not a loop-control gap, so investment belongs in what the model sees. 3. The two harnesses fail isomorphically: normalised failure-class distributions nearly overlap and several tasks fail on the identical assertion in both arms. maka's 12-task lead comes from solving more, not from failing better. Also documents three refuted hypotheses (history-reasoning replay, pruning-induced context loss, over-editing), the measured cache cost of pruning (10.8 prefix breaks/task, 100% prune-caused, ~30k tokens), and the verifier fail-closed rule behind 59% of failed f2p tests never running.
Re-ran the duplicate-read analysis against the real trajectory.json schema (tool calls use function_name/arguments, not toolName/args -- the earlier attempt matched nothing and was discarded). Keying reads by (path, line-range) instead of path alone collapses the figure by 11x: by path 3,208 dups 28.6/task 49.4% of reads by path+range 285 dups 2.5/task 4.4% of reads 91.1% of what looked like re-reading is sed window-scrolling through a large file -- normal behaviour, not waste. The 'read short-circuit' improvement is downgraded from 'potentially 10-20% of steps' to negligible. Tool-call census from this pass (Bash 13,921 / Edit 3,575 / Read 2,053 / Write 613 / Grep 149 / Glob 42) matches the independent count in the behaviour section exactly, confirming the schema parse is correct.
Astro-Han
left a comment
There was a problem hiding this comment.
P1. The reported cost-per-pass does not match the numbers beside it. The MD gives OpenCode total $12.1438, "40 passes", and $0.319574/pass, but $12.1438 ÷ 40 = $0.3036, and $0.319574 is what you get dividing by 38 (the paired-pass count). Either the pass count is wrong or the denominator is a different cohort; whichever it is, state the cost-per-pass denominator (accepted dataset vs paired) and keep it consistent across the two files and the "35.5% higher" claim (28.75% by the $0.3036 reading).
P2. The delta signs are inverted in the tables. Under the Candidate − baseline header (deepswe-1.1-deepseek-v4-flash-maka-vs-opencode.md:29-30,48) the rows read −11.13 and −12.73 pp, while Maka is higher on every row and the TL;DR says "in Maka's favor". The sibling Terminal-Bench report uses +13.48 / +9.84 pp for the leading arm. Flip the three signs.
P2. Failure cohorts drift across the analysis. The CSV is 52 passed / 59 failed / 2 unscored, but the zh file uses 59 failures at one point, 60 at three others, 61 in the later tables, then 59 again; OpenCode similarly switches between 71, 72, and 73. Declare each denominator (scored failures, non-passes, trajectory-available) in the methodology section and label the tables accordingly; do not count unscored cells as failures.
P2. The cross-benchmark conclusion is stronger than the statistics allow. "Harness advantage is task-distribution dependent" is drawn from two independent p-values (DeepSWE p=0.0759, Terminal-Bench p=0.0118); significance in one suite and not the other is not itself a significant difference. Either add a direct heterogeneity/interaction check or reword to a descriptive statement.
P2. The CSV is committed with CRLF endings, the only non-LF text file in the repo (all 2412 other tracked text files are LF, including both sibling eval CSVs). Shell pipelines and exact-match checks see verification_failed\r in the last column, and the declared SHA-256 (caeb18c7…) reproduces only from the CRLF blob. Re-commit as LF and update the hash.
P2. The zh file is a live working document, not a frozen report. It contains an open TODO section titled 未完成项 and a prioritized product-change backlog with code locations, and its line-pinned citations are already drifting (active-tool-result-prune.ts:115 is at :125 on current main). docs/README.md puts implementation plans and time-sensitive TODOs in issues, not docs/. Move the backlog and TODO list into a GitHub issue, keep the frozen-run analysis, and pin citations to a commit SHA instead of line numbers.
P3. The two files disagree on numbers for the same run: output tokens 11,629,019 / 12,791,831 (MD:73) vs 11,931,438 / 13,254,334 (zh:33), mean steps 165 / 152 (MD:76) vs 166.9 / 151.7 (zh §3.5). Reconcile or state the different measurement (telemetry-summed vs accepted final usage).
P3. A tool-surface asymmetry is disclosed only in the zh file: the Maka arm ran with MAKA_AGENT_TOOLS=false (no todowrite/webfetch) while OpenCode made 263 + 173 such calls. Benign here (allow_internet=false makes webfetch inert), but the English Frozen setup table should state it too.
Astro-Han
commented
Aug 18, 2026
/agentic_review |
Code Review by Qodo🐞 Bugs (0)📘 Rule violations (0)📎 Requirement gaps (0)Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can keep summaries lean with Finding overflow, which tucks the rest behind 'View more' |
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for publishing the experiment artifacts and methodology in a reviewable form. I reviewed exact head 8442cc41e9351c5b33237902cb454d7129720c12 against the current main integration boundary and left two additional suggestions that are separate from the existing statistical/content review. The historical exact-head checks predate one current gate, and the current e2e check is also failing, so this is not an approval. Please do push back if these documents are intentionally frozen against an older repository policy or if the remediation table is explicitly non-authoritative.
中文摘要
感谢把实验产物和方法公开成可审查文档。对照当前 main，我补充两条与既有统计/内容意见不同的建议：一条是现行 ASF header gate，另一条是脱敏修复建议的安全边界。历史绿色检查早于该 gate，当前 e2e 也失败，因此不是批准。若文档刻意冻结在旧 policy，或修复表明确不具备指导性，也欢迎直接说明。
AI-assisted review disclosure: Codex ran an independent analysis lane; Astro-Han independently verified the exact head, production path, and severity, and owns this review.
| @@ -0,0 +1,182 @@ | |||
| # DeepSWE 1.1 — DeepSeek V4 Flash: Maka vs OpenCode | |||
There was a problem hiding this comment.
[P1] Suggestion (category ① — normal integration into current main): could both new Markdown reports carry the canonical ASF source header before the title? Current scripts/asf-license-headers.mjs covers .md files and only excludes docs/eval/*.csv; current CI runs check:asf-headers unconditionally before affected tests. This file and deepswe-1.1-maka-failure-analysis.zh.md both begin directly with #, so rebasing this exact content onto main will deterministically fail the source-header gate even though this branch’s older test run predates that policy. Please add the canonical HTML-comment header to both files and rerun the current exact-head gate. If eval reports are intended to be a reviewed exclusion, please push back and point to that policy rather than adding an ad-hoc exception.
| | 优先级 | 改动 | 实测依据 | 工作量 | 风险 | 代码位置 | | ||
| |---|---|---|---|---|---| | ||
| | **P0-1** | `[a-f0-9]{40,}` 排除 40/64 位纯 hex | 消除 957 处（46.6%） | 1 行 | 低 | `redaction.ts:41` | |
There was a problem hiding this comment.
[P2] Suggestion (category ① — a maintainer following the published remediation table): could these recommendations preserve the credential-safety invariant instead of presenting three broad deletions as one-line, low-risk fixes? Current redaction tests intentionally cover ordinary 40/64-character hex values and JSON key/token fields because those shapes can be real credentials, not only Git hashes or lexer identifiers. Removing the patterns/suffix wholesale would improve this benchmark by exposing values that production currently treats as secrets. The examples in this report also show that some “qualified key” hits are source variable assignments, so the table does not establish that the remaining 20 are true credentials. I suggest framing these as evidence for a context-aware classifier, keeping positive credential regressions, and labeling the benchmark sample/review method; please push back if a separate threat model proves these shapes are never secrets in supported transcripts.
Summary
Docs-only: English write-up of DeepSWE 1.1 harness A/B run
deepseek-v4-flash-maka-vs-opencode-deepswe-full-v1(Maka vs OpenCode, both on DeepSeek V4 Flash at max reasoning effort), plus the committed per-task outcome and failure-class CSV.This is the DeepSWE counterpart to #1719 (Terminal-Bench 2.1), and it reaches a different conclusion — which is the main reason it is worth publishing.
Full report:
docs/eval/deepswe-1.1-deepseek-v4-flash-maka-vs-opencode.mdResults
Exact two-sided McNemar over the 54 discordant pairs: p = 0.0759.
The headline is a non-result, and the report says so
p = 0.0759 does not reach the conventional 0.05 threshold. On this suite the pass-rate difference between the two harnesses is not statistically distinguishable from chance. The 12.73 pp point estimate favors Maka and is the best available estimate of the effect, but a 34:20 discordant split arises by chance roughly 8% of the time under the null.
The report states this explicitly rather than leading with the point estimate:
Why the contrast with #1719 matters
Sample sizes are comparable; what differs is lopsidedness. Two same-model paired runs on two frozen suites give opposite verdicts on statistical significance — direct evidence for the scope limitation already stated in the Terminal-Bench report ("evidence about the fixed 89-task suite, not proof of a universal advantage on other task distributions").
Also note budget exhaustion was 0% on both arms here, versus 16.85% / 26.97% on Terminal-Bench. On DeepSWE the entire observed difference sits in solution quality, not deadline behavior.
Economics: the clearer signal
Both arms spent nearly identically ($12.26 vs $12.14, 1.0% apart) and cost per attempted task is effectively equal ($0.1115 vs $0.1104), but Maka converted that spend into 30% more passes — so cost per pass differs by 35.5%, driven by effectiveness rather than resource efficiency.
A non-obvious token result is documented: Maka consumed 8.6% fewer total tokens but 71.7% more uncached input tokens. Tool-result pruning rewrites conversation history and invalidates the prompt-cache prefix; since uncached input is priced 50× higher than cache hits, the two effects nearly cancel. The report frames this as an engineering trade-off, not an efficiency ranking.
Coverage disclosure
completed_with_gaps: 226/226 cells attempted, 223 model-scored, 3 unscored infrastructure cells (2 Maka, 1 OpenCode), 0 missing final usage. The paired denominator is 110 rather than 113 as a result. Exactly one Agent admission per cell — no retries, no adjudications.The report also discloses two known DeepSWE benchmark defects that place an unmeasured floor under both arms' failure counts (#17 unpassable task, #31 wildcard base-test false negatives). They apply symmetrically and should not bias the paired direction.
Verification
2 × P[Binomial(54, 0.5) ≤ 20] = 0.075905git diff --checkRepository code tests were not run because this is a documentation-only change.