Problem
A dashboard cell moves by several percent between runs with no code change, and once read 0.1680 (-83.20%) on decompress-dict / level_16_btopt / small-4k-log-lines — six times slower than libzstd, on a cell that measures 0.54 to 0.63.
The cause is which statistic the pipeline reads. Criterion takes flat samples of a few milliseconds each; scheduler interference can only ADD time to a sample, so the distribution is skewed right. On an idle host that cell spans 728.1 to 946.8 ns on our arm and 427.4 to 533.3 ns on the libzstd arm — both skewed alike, so it is the environment and not the code.
The printed point estimate is a central statistic over that tail, and run-benchmarks.sh read it while discarding the (+/- X) printed beside it. One stalled sample out of thirty is enough to drag it a long way, and nothing downstream could tell.
Evidence
Six runs of one binary, per-side spread by estimator:
|
min |
median |
mean |
| ours |
0.20% |
1.42% |
1.54% |
| libzstd |
2.00% |
1.69% |
1.81% |
The standalone binary loop over the same code gives 725 to 740 ns across six whole-process runs, agreeing with criterion's minimum and missing its consumed point estimate by four to five percent. The two harnesses already measure the same quantity; only the statistic differs.
Raising the sample count does not help: samples are not what floats, and more of them only collects more tail while costing wall-clock.
The wasm harness has the same defect from the other direction: it collects up to 200 single-operation samples and returns their median, so the two dashboard sections read different statistics.
What this does NOT fix
A ratio also carries whole-distribution shifts of an arm. In one run of six the libzstd arm sat 5% high at every quantile, minimum included. No choice of statistic removes that, which is why the per-cell spread has to be published rather than folded away. Measured on an idle host, that residual is about 2% per side.
Solution
- Set
CRITERION_HOME for the bench run and read each <dir>/new/sample.json after it. Every directory carries a benchmark.json with full_id, which is exactly the name the bencher line prints, so the join is direct. Per-sample time is times[i] / iters[i].
- Report the minimum in place of the parsed point estimate. Same field, same unit, same downstream code, so the graphs keep their shape.
- Do the same in the wasm harness, which already sorts its samples.
- Publish min, median and max per cell so a reader can weigh a number against the noise behind it, and render that where it is read: the hovered point's tooltip, the deviation list, and the generated timing table.
- Fail the run when any benchmark cannot be joined to its samples. There is no per-row fallback: a ratio pairs two implementations, so one side falling back would divide a central estimate by the other side's minimum and invent a delta.
No change to the fixtures, the sample count or the measurement budget.
Acceptance criteria
- Each side's absolute figure reproduces run to run on an idle host to within a few tenths of a percent.
- The figure agrees with a standalone timing loop over the same code and fixture.
- Every cell carries its own spread in the published payload and in what the page renders.
- Both dashboard sections report the same statistic.
- Graph shape, units and level or stage axes unchanged.
Estimate
4h
Problem
A dashboard cell moves by several percent between runs with no code change, and once read
0.1680 (-83.20%)ondecompress-dict / level_16_btopt / small-4k-log-lines— six times slower than libzstd, on a cell that measures 0.54 to 0.63.The cause is which statistic the pipeline reads. Criterion takes flat samples of a few milliseconds each; scheduler interference can only ADD time to a sample, so the distribution is skewed right. On an idle host that cell spans 728.1 to 946.8 ns on our arm and 427.4 to 533.3 ns on the libzstd arm — both skewed alike, so it is the environment and not the code.
The printed point estimate is a central statistic over that tail, and
run-benchmarks.shread it while discarding the(+/- X)printed beside it. One stalled sample out of thirty is enough to drag it a long way, and nothing downstream could tell.Evidence
Six runs of one binary, per-side spread by estimator:
The standalone binary loop over the same code gives 725 to 740 ns across six whole-process runs, agreeing with criterion's minimum and missing its consumed point estimate by four to five percent. The two harnesses already measure the same quantity; only the statistic differs.
Raising the sample count does not help: samples are not what floats, and more of them only collects more tail while costing wall-clock.
The wasm harness has the same defect from the other direction: it collects up to 200 single-operation samples and returns their median, so the two dashboard sections read different statistics.
What this does NOT fix
A ratio also carries whole-distribution shifts of an arm. In one run of six the libzstd arm sat 5% high at every quantile, minimum included. No choice of statistic removes that, which is why the per-cell spread has to be published rather than folded away. Measured on an idle host, that residual is about 2% per side.
Solution
CRITERION_HOMEfor the bench run and read each<dir>/new/sample.jsonafter it. Every directory carries abenchmark.jsonwithfull_id, which is exactly the name the bencher line prints, so the join is direct. Per-sample time istimes[i] / iters[i].No change to the fixtures, the sample count or the measurement budget.
Acceptance criteria
Estimate
4h