Uh oh!
There was an error while loading. Please reload this page.
Benchmark suite ci - #34
Merged
Merged
Conversation
There was no benchmark suite and no CI job that would notice a cost regression. Correctness tests don't help here: the answers stay right while the memory or the file size quietly doubles, which is exactly the class of defect the v0.2 work is fixing. `benchmarks/` holds a harness, five fast cases run on every PR, one larger case for a scheduled or manual run, and checked-in ceilings. Two decisions worth stating, since a flaky or meaningless gate is worse than none: Timing is never recorded as absolute seconds. Each timing case runs the same work through scipy in the same process and records the *ratio*, which cancels most of the difference between a laptop and a shared runner, and is gated loosely (4x) because it's still the noisy one. The sharp gates are the deterministic numbers: bytes per stored nonzero (1.1x) and memory allocated by an operation (2x). Memory is measured with tracemalloc, not ru_maxrss. RSS is a process-lifetime high-water mark, so an operation staying under the peak set while building its input reports zero however much it allocates -- the first version of this suite duly reported 0 MB for an operation allocating 66 MB. tracemalloc measures allocations and resets between runs. Recorded against main, the two memory cases show what the rest of v0.2 is about: a minor-axis reduction on a 4M-nonzero array allocates 66 MB, and a misaligned-direction matmul allocates 204 MB against a 16 MB array. The ceilings are therefore deliberately generous today and should be re-recorded once those fixes land; README.md says so and names the numbers. Each case runs in its own subprocess, since measurement state and JIT warm-up would otherwise leak between them. The fast set takes ~28s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d main Three new fast cases: minor-axis max/min, per-minor-index getnnz, and minor-axis selection. All three are new surface from #22/#23, and all three shipped carrying an nnz-sized temporary -- which is the best argument this gate could have for existing. The pattern this suite was built to catch reappeared in new code within the same week, unnoticed, because nothing measured it: minor_extrema_peak_mb 66.27 MB (max/min: np.repeat + ufunc.at) minor_getnnz_peak_mb 32.02 MB (np.bincount promotes int32 -> intp) minor_selection_peak_mb 62.27 MB (_select_minor, ISSUE-30) for results of length n_minor, on a 4M-nonzero array. Baselines re-recorded against merged main so the gate reflects the code it now guards. The ceilings for the first two drop by ~80x once the reduction kernels land; the third stays until ISSUE-30 is fixed. As before these gate against getting worse, not against the current numbers being good -- README.md already says so and now names all five memory cases. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Names all five memory cases with the number on main, the number with the corresponding fix, and the reason each is high -- so the generous ceilings are self-explaining rather than looking like sloppy thresholds, and so re-recording after each fix is a mechanical check rather than a judgement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drops the dead peak_rss_mb helper and cuts the prose to what the suite does and how to run it.
aarmey
approved these changes
Sep 4, 2026
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a benchmark suite and wire it into CI as a regression gate
What this adds
plus a
benchmarksjob inci.ymlrunningpython -m benchmarks.run --set fast. The fast set takes ~28s.Baselines, and the follow-up
Ceilings were recorded on
main, before the memory fixes land. That makes two of them deliberately generous today:they gate against getting worse, not against the current numbers being good.
After #30 and #32 merge, re-record with
--recordto tighten them to the fixed values.benchmarks/README.mdsays this and names the numbers so it isn't forgotten.