The same answers, five different scores
Results · Method · Problems hit · Limitations · Future work · Run it
-success){kind=icon}
A code benchmark is reported as one number. But between the model's output and that number sits an extraction step - recovering code from whatever prose and markdown the model wrapped it in - and that step is a free parameter almost nobody reports.
Here each solution is generated once, cached, then scored under five extraction strategies. Any spread is the harness's fault, because the model output never changed.
HumanEval problems 0-49, greedy decoding, one generation per (model, problem):
| Model | raw |
prompt+body |
first_fence |
all_fences |
smart |
|---|---|---|---|---|---|
| qwen2.5-coder:3b | 0.0% | 0.0% | 94.0% | 94.0% | 94.0% |
| qwen2.5:7b-instruct | 0.0% | 0.0% | 94.0% | 94.0% | 94.0% |
94 points of spread, from extraction alone.
Being precise, because the table overstates the number of independent findings: both 0%
strategies fail for the identical reason. 100 out of 100 failures in each are
SyntaxError, and the cause is the same - the models wrap answers in fences, and backticks
are not Python.
The substantive half is that prompt+body is the original HumanEval protocol, designed
for completion models. A chat-tuned model restates the whole function inside a fence, so
the concatenation is a syntax error. A harness built for completion models silently
reports chat models as incapable.
📊 Full results, per-strategy failure reasons, and the model comparison →
Under smart, all three score 94.0%. They disagree on 4 of 50 problems
(HumanEval/10, /19, /26, /38), so the identical score is not the same behaviour —
it is three different behaviours averaging to the same number.
Worth knowing before paying for a bigger model on this class of task: 4.8x the parameters bought nothing measurable.
flowchart LR
A["HumanEval<br/>0.1 MB"] --> B["generate once<br/>temperature 0, fixed seed"]
B --> C["cache on disk<br/>keyed by model + task"]
C --> D["five extraction strategies"]
D --> E["execute in a subprocess<br/>with a timeout"]
E --> F["pass@1 per strategy"]
style C fill:#16a34a,color:#fff
style F fill:#2563eb,color:#fff
Generation and scoring are separate on purpose. Adding a sixth strategy and re-scoring costs no model time at all.
🔍 The five strategies and what each one assumes →
python src/run_eval.py 50 # generate (cached) and score
pytest -q # 19 tests, no network, no model calls
Nothing about the model changed between the first row and the third. The generations are cached on disk and every strategy reads the same files. A HumanEval number published without its extraction strategy is not a measurement of the model.
The third column is the stronger half. qwen2.5-coder:14b has 4.8x the parameters of
the 3B and scores identically under all five strategies — while failing a different set
of problems. Extraction moves the score 94 points; model size moves it zero.
run_tests executes code written by a language model. It runs in a separate process, in a
throwaway temp directory, with a timeout - but it is not a security sandbox. Do not
point it at untrusted generations on a machine you care about.
The runner interpolated repr(entry_point), so HumanEval's check() received the string
"add" instead of the function. Every test failed with 'str' object is not callable -
the harness reported 0% for models that were answering correctly.
That is precisely the failure this repo exists to measure, and it happened here first.
🛠 Every problem hit while building this →
| 📊 Results | Full tables, failure reasons, model comparison |
| 🔍 Method | The five strategies, caching, execution |
| 🛠 Problems hit | The entry-point bug, plotly, buffered logs |
| ⚠ Limitations | Why 94% is not comparable to published pass@1 |
| 🚀 Future work | Full 164, pass@k, MBPP, a real sandbox |
src/core.py data loading, generation + cache, five strategies, execution
src/run_eval.py generate once, score every way, write results/scores.json
tests/ 19 tests using hand-written "model output" - no ollama needed
docs/ detailed documentation
Python 3.11+ · Ollama (local inference) · pandas · pyarrow
· pytest · ruff ·
GitHub Actions · HumanEval via Hugging Face Hub
HumanEval · pass@k · pass@1 · code generation benchmark · LLM evaluation · evaluation harness · answer extraction · benchmark reproducibility · local LLM · Ollama · Qwen2.5 · code LLM · prompt sensitivity · harness bias · LLM benchmarking methodology · deterministic evaluation
MIT - see LICENSE.