Summary
On codex/traits-2-impl (the ppvm-*-2 branch, #204 — ppvm-conformance-2 does not exist on main), tableau_surface_bench panics partway through every run. mise run perf-report treats this as a warning and continues, so three bench groups have never produced a single measurement and the perf gate silently covers less than it appears to.
Symptom
$ mise run perf-report -- --bench tableau_surface_bench --launches 4
...
warning: tableau_surface_bench exited 101 (×4, once per launch)
The panic:
crates/ppvm-conformance-2/benches/tableau_surface/mod.rs:88:5
assert_bare_eq — old-vs-new to_string() mismatch on a 96-qubit tableau
reached from the channel! macro at crates/ppvm-conformance-2/benches/tableau_surface/noise.rs:18.
Impact
Groups are registered in order in tableau_surface_bench.rs; the abort happens in noise, so everything after it never runs:
| group | runs today |
|---|
construction, display, clifford, rotation, measurement | yes (registered before noise) |
noise | aborts here |
observation, projection, sparse_amplitudes | never |
So there is no perf coverage at all for observation, projection, or sparse amplitudes, and no coverage for the part of noise after the failing fixture. Anyone reading a green-ish perf-report would reasonably assume otherwise.
Two things to fix, and they're independent:
- The fixture.
assert_bare_eq compares the old and new engines' to_string() on a 96-qubit tableau inside a benchmark setup path. Either it's a real old-vs-new divergence worth investigating on its own, or the assertion is too strict for a bench fixture and should be narrowed or dropped. - The harness.
benchmarks/perf_regression_report.py should treat a non-zero bench exit as a failure, not a warning. A benchmark binary that dies halfway through is not a partial result to summarize; silently degrading coverage is worse than a red run, because it looks like a pass.
Provenance
Pre-existing, not caused by any in-flight change: reproduced in a pristine git archive HEAD | tar -x checkout of the branch tip, so it predates the current working-tree changeset. Two separate investigations reached different conclusions about the cause (one reverting a single file and concluding "it must be one of the others"), which is itself a reason to fix the harness — a warning-level abort invites exactly that kind of misattribution.
🤖 Generated with Claude Code
Summary
On
codex/traits-2-impl(theppvm-*-2branch, #204 —ppvm-conformance-2does not exist onmain),tableau_surface_benchpanics partway through every run.mise run perf-reporttreats this as a warning and continues, so three bench groups have never produced a single measurement and the perf gate silently covers less than it appears to.Symptom
The panic:
reached from the
channel!macro atcrates/ppvm-conformance-2/benches/tableau_surface/noise.rs:18.Impact
Groups are registered in order in
tableau_surface_bench.rs; the abort happens innoise, so everything after it never runs:construction,display,clifford,rotation,measurementnoise)noiseobservation,projection,sparse_amplitudesSo there is no perf coverage at all for observation, projection, or sparse amplitudes, and no coverage for the part of
noiseafter the failing fixture. Anyone reading a green-ishperf-reportwould reasonably assume otherwise.Two things to fix, and they're independent:
assert_bare_eqcompares the old and new engines'to_string()on a 96-qubit tableau inside a benchmark setup path. Either it's a real old-vs-new divergence worth investigating on its own, or the assertion is too strict for a bench fixture and should be narrowed or dropped.benchmarks/perf_regression_report.pyshould treat a non-zero bench exit as a failure, not a warning. A benchmark binary that dies halfway through is not a partial result to summarize; silently degrading coverage is worse than a red run, because it looks like a pass.Provenance
Pre-existing, not caused by any in-flight change: reproduced in a pristine
git archive HEAD | tar -xcheckout of the branch tip, so it predates the current working-tree changeset. Two separate investigations reached different conclusions about the cause (one reverting a single file and concluding "it must be one of the others"), which is itself a reason to fix the harness — a warning-level abort invites exactly that kind of misattribution.🤖 Generated with Claude Code