Problem
CI bench output emits "Outside parity band" rows for both positive and negative shifts. Positive shifts mean we beat the donor (e.g. ratio < 1.0 = smaller output, or speed > donor) — these are wins, not regressions. They should not appear in the "outside band" alert channel.
Example bad output:
x86_64-gnu • scenario • stage:level • none • large-log-stream + stage=compress, level=level_10_lazy: 0.5054 (-49.46%) outside band
0.5054 = we are 49.46% smaller than donor on that compress scenario. This is a WIN, but the line reads as if it is a regression.
Scope
In the aggregator/report generator (likely .github/scripts/run-benchmarks.sh plus any post-processor that emits the "Outside parity band" lines), gate the alert channel on the SIGN of the delta:
- Ratio axis: only flag rows where
rust_ratio / ffi_ratio > 1.0 (we produced a larger output) — i.e. a real regression.
- Speed axis: only flag rows where
rust_speed < ffi_speed (we are slower).
- Positive shifts in our favor should either be silent OR routed to a separate "Notable wins" channel — never under the "Outside parity band" header.
Acceptance criteria
Estimate
2h (script + sign check + smoke test on existing payload).
Related
Problem
CI bench output emits "Outside parity band" rows for both positive and negative shifts. Positive shifts mean we beat the donor (e.g. ratio < 1.0 = smaller output, or speed > donor) — these are wins, not regressions. They should not appear in the "outside band" alert channel.
Example bad output:
0.5054 = we are 49.46% smaller than donor on that compress scenario. This is a WIN, but the line reads as if it is a regression.
Scope
In the aggregator/report generator (likely
.github/scripts/run-benchmarks.shplus any post-processor that emits the "Outside parity band" lines), gate the alert channel on the SIGN of the delta:rust_ratio / ffi_ratio > 1.0(we produced a larger output) — i.e. a real regression.rust_speed < ffi_speed(we are slower).Acceptance criteria
Estimate
2h (script + sign check + smoke test on existing payload).
Related