Add a reproducible benchmark harness and an advisory CI gate - #121
Merged
Conversation
There was no benchmark anywhere in the repo, and Tier 1 is worth ~3x end to end with nothing to stop it silently regressing. scripts/benchmark.py walks the 97-frame corpus and reports frames/sec. The corpus is the right workload because it exercises DNS, DHCP, GRE, VLAN tags and IPv6 extension headers rather than one synthetic frame, and frames the library rejects are counted rather than skipped, so a change that "speeds things up" by decoding less shows up. --compare times dpkt and scapy on the same frames; both live in a new bench dependency group, so the package keeps zero runtime dependencies. Absolute throughput is a property of the machine as much as the code, so a committed baseline cannot be compared against a run on a shared CI runner. Every run therefore also times a fixed calibration workload built from the same primitives as the decode path and reports a normalized figure, which is what --check compares. Measured: that collapses the gap between two interpreters on one machine from 5.5% to 0.8%. Two limits found while building it, both handled rather than hidden: - Sizing. A calibration pass that ran for barely a millisecond timed the scheduler, not the machine, and put 24% of noise straight into the gate. The calibration now runs a fixed 1000 passes, sized to tens of milliseconds, independent of --repetitions. - Interpreters. Normalization cancels machine speed but not CPython version: the same code normalizes to 6.8 on 3.12 and 7.6 on 3.13. A baseline belongs to the Python that recorded it, --check says so when they differ, and the CI job pins 3.12 to match. The CI job is advisory (continue-on-error) on purpose. Throughput on a shared runner is noisy, and a threshold picked here without runner data would start failing pull requests that changed nothing. It measures, compares and reports; setting a real threshold is a maintainer call once #103 has runner numbers to look at. Comparison output carries its own caveats, because a benchmark nobody can check is the problem this issue set out to fix: the libraries are not asked for identical work (dpkt leaves the DNS-over-TCP payload as raw bytes where this library decodes into DNSOverTCP and DNS), they materialize different amounts per frame, and it is one machine. docs/CLAIMS.md records the post-Tier-1 re-measurement: 121,945 f/s against dpkt 1.9.8's 105,292 and scapy 2.7.0's 19,657. Claim 1.2 was gated on #82/#83/#84 and is now measured — 1.16x dpkt where v1.3.0 was 2.9x slower — with the depth and single-machine caveats attached and the README wording left as a maintainer decision. Closes#86. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJnVMNGwTRDktC4rkABtgt
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes Tier 1. There was no benchmark anywhere in the repo, and the tier is worth ~3× end to end with nothing to stop it silently regressing.
scripts/benchmark.py— walks the 97-frame corpus, reports frames/sec. Frames the library rejects are counted rather than skipped, so a change that "speeds things up" by decoding less shows up.--compare— timesdpktandscapyon the same frames. Both live in a newbenchdependency group, so the package keeps zero runtime dependencies.benchmarks/baseline.json+--check— regression comparison, wired into CI.One command
Measured (post-Tier-1, CPython 3.12.3, x86-64 Linux, best of 9)
v1.3.0 measured 2.9× slower than dpkt.
docs/CLAIMS.mdclaim 1.2 was gated on #82/#83/#84 and is now measured, with caveats attached (below) and the README wording deliberately left to you.The CI job is advisory, on purpose
continue-on-error: true. Throughput on a shared runner is noisy, and a threshold I picked here without runner data would start failing pull requests that changed nothing — the one failure mode that costs you time rather than me. The job measures, compares and writes a job summary; setting a real threshold is your call once there are runner numbers to look at. This PR's own run will produce the first of them.Two limits found while building it
Both are handled in code rather than hidden, and are the reason the numbers above are worth anything:
--repetitions.--checkre-measures with the baseline's own settings, warns on a version mismatch, and the CI job pins 3.12 to matchbaseline.json.Fairness caveats ship with the output
The issue's premise is that every published benchmark in this space is unusable, so the harness prints, under every comparison:
DNSOverTCP→DNS. Where the depths differ, we are doing more..layers()).Verification
uv run ruff check/ruff format --check/mypyclean;uv run pytest— 757 passedbenchmarkjob present,continue-on-error: true, Python pinned to 3.12CHANGELOG.mdentry under## [Unreleased]Closes#86.
🤖 Generated with Claude Code
https://claude.ai/code/session_01QJnVMNGwTRDktC4rkABtgt
Generated by Claude Code