Skip to content

Add a reproducible benchmark harness and an advisory CI gate - #121

Merged
EONRaider merged 1 commit into
masterfrom
claude/decoder-depth-polish-8m54to
Sep 1, 2026
Merged

Add a reproducible benchmark harness and an advisory CI gate#121
EONRaider merged 1 commit into
masterfrom
claude/decoder-depth-polish-8m54to

Conversation

@EONRaider

Copy link
Copy Markdown
Owner

Summary

Closes Tier 1. There was no benchmark anywhere in the repo, and the tier is worth ~3× end to end with nothing to stop it silently regressing.

  • scripts/benchmark.py — walks the 97-frame corpus, reports frames/sec. Frames the library rejects are counted rather than skipped, so a change that "speeds things up" by decoding less shows up.
  • --compare — times dpkt and scapy on the same frames. Both live in a new bench dependency group, so the package keeps zero runtime dependencies.
  • benchmarks/baseline.json + --check — regression comparison, wired into CI.

One command

uv run --group bench python scripts/benchmark.py --compare

Measured (post-Tier-1, CPython 3.12.3, x86-64 Linux, best of 9)

frames/secvs. us
netprotocols121,945
dpkt 1.9.8105,2920.86×
scapy 2.7.019,6570.16×

v1.3.0 measured 2.9× slower than dpkt. docs/CLAIMS.md claim 1.2 was gated on #82/#83/#84 and is now measured, with caveats attached (below) and the README wording deliberately left to you.

The CI job is advisory, on purpose

continue-on-error: true. Throughput on a shared runner is noisy, and a threshold I picked here without runner data would start failing pull requests that changed nothing — the one failure mode that costs you time rather than me. The job measures, compares and writes a job summary; setting a real threshold is your call once there are runner numbers to look at. This PR's own run will produce the first of them.

Two limits found while building it

Both are handled in code rather than hidden, and are the reason the numbers above are worth anything:

  1. Calibration sizing. Absolute throughput is a property of the machine, so each run also times a fixed calibration workload and reports a normalized figure — that collapses the gap between two interpreters on one machine from 5.5% to 0.8%. But my first version ran the calibration for barely a millisecond, which timed the scheduler and put 24% of noise straight into the gate. It now runs a fixed 1000 passes, sized to tens of milliseconds, independent of --repetitions.
  2. Normalization does not cancel the interpreter. The same code and machine normalize to 6.8 on CPython 3.12 and 7.6 on 3.13. A baseline belongs to the Python that recorded it: --check re-measures with the baseline's own settings, warns on a version mismatch, and the CI job pins 3.12 to match baseline.json.

Fairness caveats ship with the output

The issue's premise is that every published benchmark in this space is unusable, so the harness prints, under every comparison:

  • The libraries are not asked for identical work — on the DNS-over-TCP frames dpkt stops at TCP and leaves the payload as raw bytes where netprotocols continues into DNSOverTCPDNS. Where the depths differ, we are doing more.
  • Each materializes a different amount per frame (frozen dataclasses vs. lazy attributes vs. forced .layers()).
  • One machine, one corpus, one CPython — quote the machine with the number.

Verification

  • uv run ruff check / ruff format --check / mypy clean; uv run pytest — 757 passed
  • Workflow YAML parses; benchmark job present, continue-on-error: true, Python pinned to 3.12
  • CHANGELOG.md entry under ## [Unreleased]

Closes#86.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QJnVMNGwTRDktC4rkABtgt


Generated by Claude Code

There was no benchmark anywhere in the repo, and Tier 1 is worth ~3x
end to end with nothing to stop it silently regressing.
scripts/benchmark.py walks the 97-frame corpus and reports frames/sec.
The corpus is the right workload because it exercises DNS, DHCP, GRE,
VLAN tags and IPv6 extension headers rather than one synthetic frame,
and frames the library rejects are counted rather than skipped, so a
change that "speeds things up" by decoding less shows up. --compare
times dpkt and scapy on the same frames; both live in a new bench
dependency group, so the package keeps zero runtime dependencies.
Absolute throughput is a property of the machine as much as the code,
so a committed baseline cannot be compared against a run on a shared CI
runner. Every run therefore also times a fixed calibration workload
built from the same primitives as the decode path and reports a
normalized figure, which is what --check compares. Measured: that
collapses the gap between two interpreters on one machine from 5.5% to
0.8%.
Two limits found while building it, both handled rather than hidden:
- Sizing. A calibration pass that ran for barely a millisecond timed
the scheduler, not the machine, and put 24% of noise straight into
the gate. The calibration now runs a fixed 1000 passes, sized to
tens of milliseconds, independent of --repetitions.
- Interpreters. Normalization cancels machine speed but not CPython
version: the same code normalizes to 6.8 on 3.12 and 7.6 on 3.13.
A baseline belongs to the Python that recorded it, --check says so
when they differ, and the CI job pins 3.12 to match.
The CI job is advisory (continue-on-error) on purpose. Throughput on a
shared runner is noisy, and a threshold picked here without runner data
would start failing pull requests that changed nothing. It measures,
compares and reports; setting a real threshold is a maintainer call
once #103 has runner numbers to look at.
Comparison output carries its own caveats, because a benchmark nobody
can check is the problem this issue set out to fix: the libraries are
not asked for identical work (dpkt leaves the DNS-over-TCP payload as
raw bytes where this library decodes into DNSOverTCP and DNS), they
materialize different amounts per frame, and it is one machine.
docs/CLAIMS.md records the post-Tier-1 re-measurement: 121,945 f/s
against dpkt 1.9.8's 105,292 and scapy 2.7.0's 19,657. Claim 1.2 was
gated on #82/#83/#84 and is now measured — 1.16x dpkt where v1.3.0 was
2.9x slower — with the depth and single-machine caveats attached and
the README wording left as a maintainer decision.
Closes#86.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJnVMNGwTRDktC4rkABtgt
@EONRaider
EONRaider merged commit 83bab02 into masterSep 1, 2026
7 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a reproducible benchmark harness and a CI regression gate

2 participants

@EONRaider@claude