Skip to content

Ship decode_frame(), the chain walker - #126

Merged
EONRaider merged 1 commit into
masterfrom
claude/decoder-depth-polish-8m54to
Sep 2, 2026
Merged

Ship decode_frame(), the chain walker#126
EONRaider merged 1 commit into
masterfrom
claude/decoder-depth-polish-8m54to

Conversation

@EONRaider

Copy link
Copy Markdown
Owner

Summary

The README taught a hand-rolled eight-line loop, ARCHITECTURE.md showed it
again, and the test suite kept a private copy — so the library's most-used
function was the one function it did not provide.

fromnetprotocolsimportdecode_framepacket=decode_frame(frame) # Packet(Ethernet(...), IPv4(...), TCP(...))frame[packet.consumed:] # whatever the chain did not decode

Closes#88.

What's included

src/netprotocols/walk.pydecode_frame(), with the four things a
copy-pasted loop never has.

  • An explicit starting layer.decode_frame(buf, start=IPv4) for a buffer
    that begins mid-stack — a tunnel payload, a packet quoted inside an ICMP
    error, a non-Ethernet link type. There was previously no way to ask for this.
  • Bounded depth.max_depth (default 32) raises the new
    MaxDepthExceededError, rooted at ProtocolError like everything else.
  • A lax mode.lax=True ends the walk on a ProtocolError and returns the
    layers decoded so far, with the reason on packet.stopped_by.
  • Per-call decoder overrides.decode_as={"udp.port": {6969: DNS}}, built
    on A public protocol registry #87's Registry.derive().

Packet gains stopped_by (why a walk ended early; None for a packet you
built) and consumed (bytes its headers occupy). Both default to the
constructed-packet values, so Packet(eth, ip) is unchanged.

Copies retired — README.md, ARCHITECTURE.md, tests/test_corpus.py (now a
thin adapter), tests/test_contract.py (inline copy gone).

Teststests/test_walk.py, 85 new tests.

Verification

  • uv run ruff check and uv run ruff format --check are clean
  • uv run mypy is clean (strict)
  • uv run pytest passes locally — 882 passed, coverage 99.88% (gate 98%)
  • CHANGELOG.md has an entry under ## [Unreleased]

New protocol or dispatch change — also:

  • Decode contract in src/netprotocols/_base.py unchanged; next_protocol() gains an optional parameter, covered below
  • No new EtherType / IPProtocol numbers
  • No new protocol classes, so tests/test_fuzz.py::ALL_PROTOCOLS is unchanged
  • No new fixtures — the walker is exercised over the existing corpus
  • README.md and ARCHITECTURE.md both updated (this issue's criterion is that they stop teaching the loop)

Notes

I was wrong in #87 about this being a small change

I said derive() would make decode_as "a small change rather than a second
dispatch mechanism." The derive() part held; the rest did not. next_protocol()
takes no arguments and reads the process-wide tables directly, so a walker
holding a custom registry had no way to redirect it — and decision 1 on #87
explicitly ruled out changing that signature.

The resolution is an optionalregistry=None parameter, passed to the same
dispatch helpers as before. That matters because the helpers already own their
table names, so nothing is duplicated and there is no second dispatch path to
keep in sync. Every existing zero-argument call is unaffected.

Cost, measured in one process (identical bodies, one with the parameter):

no param 72.5 ns
optional param 74.6 ns (+2.1)

+2.1 ns per dispatch, roughly 0.1% of a frame decode. A first cross-run
comparison suggested +15 ns; that was the machine noise documented in #87
(±10–15% between consecutive runs), not the parameter. The in-process A/B is
the number that resolves.

If you'd rather this signature stayed frozen, the alternative is dropping
decode_as from #88 and reopening Q4 — say so and I'll do that instead.

memoryview: benchmarked, then rejected

The issue said to benchmark rather than assume, so I did. Wrapping each frame in
a memoryview measures 0.95× on the corpus — 5% slower — because for a
single small frame the view costs more to build than the copy it saves. That
matches the warning in #100.

So the walker slices whatever it is handed and never converts: bytes stays
fastest for one frame, and a memoryview over a large contiguous capture buffer
keeps slices zero-copy, which is #100's 1.8× case. Converting internally would
have been worse than either. Byte-exact round-tripping through a memoryview is
asserted over the corpus.

What the depth guard actually saves

Not a runaway chain — every header validates its own declared length, so chains
already terminated. The case it saves is a zero-length header that chains to
itself
: the cursor never advances and the walk never ends. Nothing in the
decode contract forbids that, and since #87 a third-party registration can
produce it. tests/test_walk.py::TestHostileChains builds exactly that and
asserts the walker returns rather than hangs.

Two deliberate exclusions

scripts/benchmark.py keeps its raw loop. It measures decode throughput
against a committed baseline; wrapping it in Packet construction would change
what the gate measures and invalidate benchmarks/baseline.json. Worth
revisiting as a separate --walker comparison.

#92's decode_lax() is not in here.#92 asks that its lenient mode and this
one "share one concept, not invent two", and it wants the stop reason to be
#91's structured diagnostic. So stopped_by holds the ProtocolError itself
rather than a new parallel type — when #91 gives those exceptions structure,
this contract gains it for free with no API change.

On the README

Touching it was this issue's acceptance criterion, so I did — but only the
decoding section, plus a short new section on registering a protocol we don't
ship (which #87 left undocumented anywhere user-facing). The coverage table and
everything comparative are untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QJnVMNGwTRDktC4rkABtgt


Generated by Claude Code

The README taught a hand-rolled eight-line loop, ARCHITECTURE.md showed
it again, and the test suite kept a private copy, so the library's
most-used function was the one function it did not provide. Everyone
using it maintained their own.
decode_frame(frame) returns a Packet of every decoded layer, with the
parts a copy-pasted loop never has.
An explicit starting layer. decode_frame(buf, start=IPv4) walks a
buffer that begins mid-stack: a tunnel payload, a packet quoted inside
an ICMP error, a non-Ethernet link type. There was previously no way to
ask for this at all.
Bounded depth. max_depth (default 32) raises the new
MaxDepthExceededError, rooted at ProtocolError like everything else.
Chains already terminated, since every header validates its own
declared length against the buffer, so this bounds cost rather than
correctness. The case it actually saves is a zero-length header that
chains to itself, which nothing in the decode contract forbids and a
third-party registration can now produce: the cursor never advances and
the walk never ends. tests/test_walk.py builds exactly that and asserts
the walker returns instead of hanging. The corpus peaks at 5 layers.
A lax mode that reports instead of raising. lax=True ends the walk on a
ProtocolError and returns the layers decoded so far, with the reason on
packet.stopped_by, which is what a capture tool needs when frame
4,000,001 is malformed. It relaxes the walk, never a decoder: every
layer it returns was decoded under the ordinary strict rules, and that
is asserted. Caller mistakes -- a bad max_depth, an unknown decode_as
table -- are not absorbed by it.
Per-call decoder overrides. decode_as={"udp.port": {6969: DNS}} reads
DNS on a nonstandard port for one call without touching global state,
built on Registry.derive(). Making that work needed next_protocol() to
be redirectable, and it read the process-wide tables directly. It now
takes an optional registry, passed to the same dispatch helpers as
before, so no table knowledge is duplicated and the default path stays
a single dict.get. Measured in one process, the optional parameter
costs +2.1 ns per dispatch, about 0.1% of a frame decode; every
existing zero-argument call is unaffected.
Packet gains stopped_by (why a walk ended early; None for a packet you
built) and consumed (the bytes its headers occupy, so
frame[packet.consumed:] is what the chain did not decode). Both default
to the constructed-packet values, so Packet(eth, ip) is unchanged.
On memoryview, which the issue asked to benchmark rather than assume:
wrapping each frame in one measures 0.95x on the corpus -- 5% slower --
because for a single small frame the view costs more to build than the
copy it saves. So the walker slices whatever it is handed and never
converts. bytes stays fastest for one frame, and a memoryview over a
large capture buffer keeps slices zero-copy, which is the case worth
1.8x once a pcap reader exists. Byte-exact round-tripping through a
memoryview is asserted over the corpus.
The copies are retired: README.md and ARCHITECTURE.md now show the
shipped call, tests/test_corpus.py's walk() is a thin adapter over it,
and test_contract.py's inline copy is gone. One test still walks the
corpus with a hand-rolled loop and asserts the shipped walker agrees
layer for layer -- that is what makes retiring the others safe.
scripts/benchmark.py keeps its raw loop deliberately: it measures
decode throughput against a committed baseline, and wrapping it in
Packet construction would change what the gate measures.
Closes#88.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJnVMNGwTRDktC4rkABtgt
@EONRaider
EONRaider merged commit a61f9b4 into masterSep 2, 2026
7 checks passed
This was referenced Sep 4, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ship decode_frame() — the chain walker belongs in the library

2 participants

@EONRaider@claude