perf: load test every implemented protocol version, fix readUUID

[mcollina]

Targets compatibility (#338), because it exists to validate that branch's codecs.

Why

#338 taught Base[kGetApi] to negotiate down to Produce v3 and Fetch v4. Every broker in CI negotiates to the newest version, so those legacy codecs are covered for correctness by test/integration/*.compat-test.ts and never for speed. If one were accidentally quadratic, allocated per record, or defeated a fast path in DynamicBuffer, nothing in the repo would notice.

Design

The experiment pins the client codec against a fixed modern broker, so the protocol version is the only variable. pinApiVersions() already existed in test/helpers/api-versions.ts; a Kafka 4.x broker still accepts every version this package implements (KIP-896 set the floor at exactly our lowest codecs), so the whole matrix runs against one broker.

Running against the 1.1.0 stack instead would vary the JVM, the storage engine, the on-disk format and the codec at once — that can't attribute a difference to our code. It runs as an explicitly labelled sanity check (tier 2), not a measurement.

Three tiers: codecs with no broker, pinned versions against a live broker, and the legacy-broker sanity run. scripts/run-protocol-load-test.sh runs them in the right order.

Verdict

The legacy codecs are not slower. Across 9 Produce versions, 14 Fetch versions, three payload shapes and two acks settings, no legacy version is consistently slower than the newest. Where a reproducible difference exists it runs the other way.

	Produce v3–v11	Fetch v4–v11	Fetch v12–v17
CPU µs/msg (1 MB fetches)	4.00–4.57, no ordering	1.39–1.42	1.72–1.76

Against real Apache Kafka 1.1.0 the client negotiates Produce v5 / Fetch v7 unaided and matches the modern broker to within 1% (1.40 vs 1.39 µs/msg).

Findings — both in the newest versions

1. Reader.readUUID was 14x slower than necessary — fixed here. It hyphenated with a capture-group regex on every call: 801 ns against 56 ns for the equivalent slicing. Fetch v13+ identifies topics by UUID, so this made the newest Fetch versions 15–18% slower to decode than the name-based ones. This is the only src/ change in the PR. 1653 protocol and codec tests pass unchanged.

2. Flexible framing costs ~23% per message at 1 MB fetch responses. The break is exactly at the v11→v12 boundary, where compact collections and tagged fields begin. Four consecutive runs, tight clusters either side. Not fixed and the mechanism is not isolated — it is recorded rather than guessed at. It matters because v17 is what the client negotiates by default against a modern broker.

Also noted, version-independent so invisible to this comparison: the shared record encoder costs ~48% more per record at 10,000 records than at 100 (1898 → 5418 ns/record out to 400k). Roughly 1.3x per 10x, so allocation and cache pressure — createRecord allocates a Writer per record — not anything quadratic. It is the largest single lever on producer throughput found here.

What the measurement cannot support

Recorded in LOAD_TESTING.md rather than glossed over. Tier 1 Fetch at maxBytes=4096 and tier 0 Fetch decode both have a ~20% noise floor on the machine this ran on, well above the 15% threshold the criteria use. The verdict rests on tier 1 at 64 KB and 1 MB and on Produce, where clusters are tight across four runs and three shuffle seeds. Making tier 0 Fetch decode reliable would mean running each version in its own process.

Methodology corrections

Three things that first presented as findings about the client and were not. They are documented because anyone re-running this will hit them:

1. Fixed record timestamps deleted the test data. The generator stamped every record 1700000000000 (Nov 2023) for byte reproducibility. Kafka applies retention by the largest record timestamp in a segment, so the broker deleted the seeded log mid-sweep and consumers correctly read an empty partition — which looked exactly like an intermittent consumer stall. Base is now Date.now() captured once; byte reproducibility survives because a batch stores its base as a fixed-width INT64 and each record as a varint delta.
2. A constant shuffle seed confounded version with position. Fetch v5 measured 5.79 / 6.01 / 5.97 at maxBytes=4096 across three runs — entirely because it was always first. The seed is now PROTOCOL_BENCH_SEED, and varying it is documented as mandatory before believing any single-version result.
3. Warmup must match workload shape. V8 tiers up the many-small-fetches path separately from the few-large-fetches path, so one global warmup left the first cells of each mode paying for it.

Validity guards

• Zero broker-side message conversions across every implemented version of both APIs, verified over JMX by guards.ts. Without this the sweeps could be measuring the broker's down-conversion cost under the client's label.
• Every cell asserts the negotiated version equals the pin. A silently unpinned run would report the newest codec's numbers under an old version's label — the one failure mode here that yields a confident wrong answer.
• The fourteen Fetch response layouts come from one parameterised builder driven by five traits off the schemas; it is self-checking, parsing what it writes with the real codec and asserting the record count survives.

Not included

No CI job. Tier 0 needs no Docker and could reasonably run in CI later, but the live tiers need pinned CPUs to mean anything and would be noise in a shared runner.

Test plan

• npm run lint and npm run typecheck clean
• 1653 tests pass across test/apis/*/*.test.ts and test/protocol/*.test.ts, covering the readUUID change
• Full sweep: ./scripts/run-protocol-load-test.sh

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kj49eYmNCELDfrP2xa1YpA