Skip to content

fix(ci): benchmark timings are a central statistic over a scheduler-skewed sample distribution #514

Description

@polaz

Problem

A dashboard cell moves by several percent between runs with no code change, and once read 0.1680 (-83.20%) on decompress-dict / level_16_btopt / small-4k-log-lines — six times slower than libzstd, on a cell that measures 0.54 to 0.63.

The cause is which statistic the pipeline reads. Criterion takes flat samples of a few milliseconds each; scheduler interference can only ADD time to a sample, so the distribution is skewed right. On an idle host that cell spans 728.1 to 946.8 ns on our arm and 427.4 to 533.3 ns on the libzstd arm — both skewed alike, so it is the environment and not the code.

The printed point estimate is a central statistic over that tail, and run-benchmarks.sh read it while discarding the (+/- X) printed beside it. One stalled sample out of thirty is enough to drag it a long way, and nothing downstream could tell.

Evidence

Six runs of one binary, per-side spread by estimator:

min median mean
ours 0.20% 1.42% 1.54%
libzstd 2.00% 1.69% 1.81%

The standalone binary loop over the same code gives 725 to 740 ns across six whole-process runs, agreeing with criterion's minimum and missing its consumed point estimate by four to five percent. The two harnesses already measure the same quantity; only the statistic differs.

Raising the sample count does not help: samples are not what floats, and more of them only collects more tail while costing wall-clock.

The wasm harness has the same defect from the other direction: it collects up to 200 single-operation samples and returns their median, so the two dashboard sections read different statistics.

What this does NOT fix

A ratio also carries whole-distribution shifts of an arm. In one run of six the libzstd arm sat 5% high at every quantile, minimum included. No choice of statistic removes that, which is why the per-cell spread has to be published rather than folded away. Measured on an idle host, that residual is about 2% per side.

Solution

  1. Set CRITERION_HOME for the bench run and read each <dir>/new/sample.json after it. Every directory carries a benchmark.json with full_id, which is exactly the name the bencher line prints, so the join is direct. Per-sample time is times[i] / iters[i].
  2. Report the minimum in place of the parsed point estimate. Same field, same unit, same downstream code, so the graphs keep their shape.
  3. Do the same in the wasm harness, which already sorts its samples.
  4. Publish min, median and max per cell so a reader can weigh a number against the noise behind it, and render that where it is read: the hovered point's tooltip, the deviation list, and the generated timing table.
  5. Fail the run when any benchmark cannot be joined to its samples. There is no per-row fallback: a ratio pairs two implementations, so one side falling back would divide a central estimate by the other side's minimum and invent a delta.

No change to the fixtures, the sample count or the measurement budget.

Acceptance criteria

  • Each side's absolute figure reproduces run to run on an idle host to within a few tenths of a percent.
  • The figure agrees with a standalone timing loop over the same code and fixture.
  • Every cell carries its own spread in the published payload and in what the page renders.
  • Both dashboard sections report the same statistic.
  • Graph shape, units and level or stage axes unchanged.

Estimate

4h

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions