Skip to content

docs: publish Ollama DeepSeek Flash harness comparison - #1916

Draft
Astro-Han wants to merge 1 commit into
mainfrom
docs/eval-ollama-0731-comparison
Draft

docs: publish Ollama DeepSeek Flash harness comparison#1916
Astro-Han wants to merge 1 commit into
mainfrom
docs/eval-ollama-0731-comparison

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

Summary

  • Publishes the audited 89-task Maka versus OpenCode run on Ollama Cloud deepseek-v4-flash:0731, including the per-task outcome projection.
  • Combines this result with the previous DeepSeek V4 Flash run using a task-clustered exact sign-flip test: Maka leads 113 to 92 across the two runs, with p = 0.00416.
  • Separates effectiveness from economics: the current run is 52 to 43, while OpenCode uses fewer tokens on the 88 fully metered pairs.
  • Discloses the one non-recoverable final-usage gap, budget-exhaustion shift, provider-condition difference, and full infrastructure audit. No harness code changes are included.

Follow-up to #1719.

Verification

  • git diff --cached --check
  • Recomputed the committed CSV aggregates: 89 tasks, Maka 52, OpenCode 43, 18 Maka-only, 9 OpenCode-only, 25 versus 18 budget exhaustions.
  • Recomputed the two-run task-clustered exact result from both committed CSVs: 21-outcome lead, p = 0.004163928097113967.
  • Verified the committed CSV SHA-256 and all relative Markdown links.
  • Audited all 178 final Harbor results, 176 CTRF reports, both missing-CTRF traces, verifier retry evidence, and OpenCode provider telemetry; no accepted infrastructure failure was found.
  • Code tests were not run because this PR changes only Markdown and CSV evidence.

@github-actionsgithub-actionsBot added the effort/M Under 500 readable lines label Aug 27, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/MUnder 500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han