b+tree: state the node geometry, and make the benchmark able to resolve what it judges - #163
b+tree: state the node geometry, and make the benchmark able to resolve what it judges#163psiha wants to merge 11 commits into
Conversation
…idden Two things a caller that measures this tree cannot currently get at. node_size is a build-time knob but it is not the number that governs anything: how many values a node holds follows from it, the header size and sizeof( Key ), and that count is what both the occupancy work and the linear-vs-binary intra-node search dispatch are keyed on. max_values_per_leaf/inner expose it. linear_search_byte_limit was a bare constant with no way to vary it, so the one experiment that separates the two search implementations - build the same tree twice, once each way - required editing the header. It now takes -DPSI_VM_LINEAR_SEARCH_BYTE_LIMIT, and 0 turns the linear path off everywhere, which is what AArch64 already does.
…udge The benchmark is now the thing that decides node geometry and search-dispatch questions, and as written it could not carry that weight at the ~15% margins those questions turn on. * EXPECT_EQ was inside the timed lookup loop - gtest machinery, once per key, 7.65M times per arm. The loop now accumulates a checksum and the checksum is verified after the clock stops; it is also what keeps the searches live. * The seed came from random_device, so two separately built A/B arms shuffled differently. It is fixed, and -DPSI_VM_BENCH_SEED varies it deliberately. * Per-key figures were a total divided by millions truncated to whole nanoseconds. They are fractional now. * A lookup pass is run three times and the best reported; one pass does not separate from scheduling noise. And the run now states the geometry and the dispatch it measured - node size, values per leaf and per inner node, the byte limit, and whether each level ended up linear or binary. Without that a build's numbers are not comparable with any other build's, and a flag that silently failed to take reads exactly like a refuted hypothesis.
Every number this benchmark produces comes from std::less on the keys themselves, which is the one case where a comparison is free: the scan already has the key in the cache line it is walking. A real index comparator does not look like that - it stores row indices and orders them by dereferencing a column, so every comparison is a scattered load. That is the whole economics of the intra-node search, and it is why a node-size or linear-vs-binary answer obtained with std::less does not transfer. So there is now a second arm with an indirect comparator, printing its own geometry and dispatch banner. Measured at 512-byte nodes it costs roughly four times the direct arm per operation, which is the size of the effect being left out. The linear path is reachable with such a comparator, and this is not incidental: is_simple_comparator says whether == may replace the double-negation equivalence test - a statement about ordering semantics, not about cost - so an indirect comparator satisfies it, and consumers specialise it for exactly this shape.
The two shipping geometries are 512 bytes and page_size, and for a 4-byte key that is 123 values per leaf against 1019 - an order of magnitude apart in the count that actually governs both the occupancy behaviour and the linear-vs-binary intra-node search dispatch. Everything between them is unreachable, so a sweep cannot say where either crossover sits; the only two data points available are the two shipping sizes, and they disagree. -DPSI_VM_BT_NODE_SIZE=n makes those sizes reachable. It overrides both existing branches and changes no default.
What the three knobs already turned upThree arms differing only in 512-byte nodes (leaf 123 values, inner 61),
4096-byte nodes (leaf 1019, inner 509) — the leaf is binary in every arm, so the variable is the Three things follow.
So the two ISAs' crossovers sit in disjoint bands — x64's in (492, 2036] bytes, AArch64's in I have not changed any default here. The gap between the two shipping node sizes is an order of One number worth flagging separately: |
|
Correction to the AArch64 half of the previous comment — I published it with a control I should have The three arms execute serially in a fixed order within each block, so on a box that drifts So: hold the 29% insert claim. The run now reverses the arm order on alternate blocks; if an Nothing about the x64 findings changes: at 512 bytes linear wins by 9–23%, and at 4096 the 509-key One more caveat on a number in that table: the |
|
The control ran, and it clears the AArch64 result rather than killing it. Same three arms, but the order reversed on alternate blocks. If an arm's number depended on when it
It does not — each arm reproduces to about 0.1–2% wherever it runs. So this workload has no position
So the shipped AArch64 default ( The second x64 box also reproduced the 4096 result — 232.1 with a linear inner against 219.7 / 214.3 Net: three ISAs, two geometries, and no single value of one global constant is right for all of them. |
The node-size sweep, and the case for separating "cheap to call" from "simple"
Direct comparator (
So for a direct comparator the crossover sits around 2048-byte nodes, and at 2048 it is close enough The consistent surprise is Indirect comparator — one that stores row indices and orders them by dereferencing a column, which
Binary is 1.45x faster at 1024 and 2.25x at 2048, and the gap grows with node size — which is what That points at something structural rather than a constant to retune. Still no default changed in this PR. What the numbers argue for is a b+tree-local threshold plus an One caveat I would rather state than bury: the AArch64 half of the sweep is not usable. Those |
…decidable The linear-vs-binary threshold is expressed in bytes, and every measurement of it so far used 4-byte keys. At a fixed node size the byte size and the value count then move together, so a crossover found that way fits "the limit is about bytes scanned" and "the limit is about number of comparisons" equally well - the data cannot tell them apart, and which one is true decides whether the constant is right for anything but int. Doubling the key width holds the node's byte size and halves the value count. At 1024-byte nodes that is 251 values / 1004 B against 125 values / 1000 B. Running the same tree at both widths across the node sizes therefore puts the crossover at the same node size if the unit is bytes, and at twice the node size if it is values. The two answers are expected to differ by comparator, which is the other half of the argument: a scan over keys the node already holds costs cache lines touched, while an indirect comparison costs one scattered load per value regardless of how wide the key is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… per regime `linear_search_byte_limit` carries three empirical claims in its comment - a value (a crossover "between 1 and 4 KiB"), a unit (bytes, not element count, "the same BYTE size for 32-bit and 64-bit keys"), and a platform exception (AArch64 disables the linear path outright because a branchless binary search "wins at EVERY size"). None of the three is checked anywhere, and the constant is consumed by every sorted container in the library. This times the two primitives directly across lengths that bracket the crossover, for uint16/uint32/uint64/float/double, and prints where they actually cross plus what the shipped constant would have picked - so a mismatch is visible rather than inferred. It measures TWO regimes, because the regime turns out to matter more than the length. A resident range is searched with every line already present, which is the best case for a binary search - its scattered probes never miss - and the worst for a linear scan, which has nothing to overlap its mispredicted exit branch with. A b+tree node is the opposite: it has just been pointer-chased to, so a linear scan streams it under the prefetcher while a binary search issues dependent misses. The two regimes disagree about which search wins, so a threshold calibrated on one of them says nothing about the other. The accumulator is written through a volatile. Checking it with EXPECT_GE( acc, 0u ) instead is trivially true for an unsigned, and the compiler duly proved the accumulator dead and deleted the entire timed loop - every figure came out 0.00 ns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pacity Behind -DPSI_VM_BT_RUNTIME_DISPATCH, default off: this is the hot path of every consumer and it wants its own measurement before it becomes the default. The compile-time form picks the search strategy from the length a node COULD have. Nodes are rarely full, so it can pick the strategy for a fill the node never reaches - and it cannot distinguish a half-full 4096-byte leaf, which scans 510 values, from a packed one that scans 1019. lookup.hpp already has the runtime form, which tests the actual length. The reason this is not simply a branch added to the hot path: the BOOST_ASSUME( num_vals <= maximum_values ) already sitting above the dispatch lets the compiler discharge the runtime comparison whenever the whole node fits under the threshold, so the small-node case keeps the branchless code it has today and only the geometries where fill genuinely varies pay for the test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
absl::btree decides the same question this tree does, and its rule is "if the key is arithmetic and the comparator is std::less or std::greater, choose linear, otherwise binary" - linear for EVERY arithmetic key, with no size condition at all. Their own comment flags the gap: "TODO(ezb): Might make sense to add condition(s) based on node-size." They also provide the means to check it: a comparator may opt in or out of linear node search through a member typedef, so the same container can be built both ways and compared against itself rather than against ours - which removes every difference except the one under test. The reference checksum is summed explicitly rather than with std::accumulate and a uint64 init: over a vector<float> that promotes the ACCUMULATOR to float, so the reference is computed in float precision and diverges from the uint64 total the lookup builds as soon as the running sum passes 2^24. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… measurement
The limit was in bytes because the crossover had been found at the same byte
size for 32- and 64-bit keys. Measured again with both widths in a real tree
(bp_tree.benchmark_key_width), that does not hold: the crossover sits between
~250 and ~510 VALUES for 4-byte keys and, separately, for 8-byte keys - which is
1004-2028 bytes in one case and 2024-4072 in the other. In bytes, one constant
means two different policies for two widths of the same type; in values it means
one. So the limit is now 256 values.
What that changes, and it is exactly what the measurement asked for:
* 4096-byte nodes, 4-byte keys: an inner node holds 509 values = 2036 bytes -
TWELVE bytes under the old 2048 limit - so it scanned linearly. Measured,
making it binary is 14% faster on lookup and 7-12% on insert. 509 > 256, so
it is binary now.
* 2048-byte nodes, 4-byte keys: a leaf holds 507 values = 2028 bytes, likewise
just under the old limit, and likewise measured as wanting binary (5.3%).
* Everything else measured keeps the search it already had: 512- and
1024-byte nodes at either width, and 2048-byte nodes at 8-byte keys, where a
linear scan is measured 21% ahead.
AArch64 stays at 0, and that is no longer an assumption: clang emits a genuinely
branchless binary search there (`fcmp; csel; csel; cbnz`) and it wins at every
length and every key type measured, floating point included. x86-64 gets a
BRANCHY loop for both integers and floating point, and that misprediction is the
only reason a linear scan competes there at all.
The header now carries the two caveats a future re-tune needs: floating-point
keys behave differently from integers on x86-64 and this single constant cannot
express that, and the isolated and in-tree numbers differ by an order of
magnitude because they are different cache regimes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The measurements separate two axes and the constant only expressed one. On x86-64 integers and floating point cross at very different lengths - measured in isolation, integers at 8 values and float/double together at 96 (Xeon 8581C) / 192 (Zen 5) / 256 (Arrow Lake) - so one number for both was picking the integer answer and applying it to float. The limit is now a variable template on the key type: 256 values for integral keys, 128 for floating-point, 0 on AArch64 for both. Also corrects the rationale in the header, which was wrong. It claimed branch misprediction is why a linear scan competes on x86-64. It is not: clang-cl emits a BRANCHLESS integer binary search on Windows (`cmovae`, where libstdc++ on the same ISA emits `cmp;jae`) and a linear scan still won in-tree there up to ~250-510 values. What keeps the scan competitive inside a b+tree is that a node has just been pointer-chased to and is cold, so the scan streams it under the prefetcher while binary search issues dependent misses - which is also why the same code on a RESIDENT array reverses the verdict, and why these numbers may only be re-tuned from an in-tree measurement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The claim that the search threshold is denominated in values rather than bytes rested on two widths, and two points cannot tell "a number of values" from "a number of bytes that happens to line up". The isolated sweep now covers uint8_t as well, and a narrow key needs care to measure at all: the generator strides by 3, so uint8_t tops out at 84 values and every longer range would WRAP - yielding an unsorted range and a silently wrong answer rather than an obviously wrong one. Lengths a type cannot represent are skipped. The in-tree sweep gains a 16-bit arm. It is the discriminating one: at 1024-byte nodes a uint16 leaf holds ~509 values but only ~1018 bytes, so the two denominations predict opposite winners there. A unique tree holds at most 65536 uint16 keys, so it is necessarily small and shallow and its absolute timings are not comparable with the 32- and 64-bit arms - it is there for the geometry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stacked on #161.
The b+tree benchmark has become the thing that decides node-geometry and
search-dispatch questions. It could not carry that weight: the differences those
questions turn on are on the order of 15%, and the harness had more distortion
than that in it.
Library
max_values_per_leaf()/max_values_per_inner().node_sizeis the knob,but it is not the number anything is keyed on — how many values a node holds
follows from it, the header size and
sizeof( Key ), and that count is whatboth the occupancy work and the linear-vs-binary intra-node search dispatch
turn on. It was not reachable from outside.
linear_search_byte_limittakes-DPSI_VM_LINEAR_SEARCH_BYTE_LIMIT. It was abare
inline constexpr, so the one experiment that separates the two searchimplementations — build the same tree twice, once each way — meant editing a
header.
0turns the linear path off everywhere, which is what AArch64already does by default.
Benchmark
EXPECT_EQwas inside the timed lookup loop: gtest machinery, once perkey, 7.65M times per arm. It now accumulates a checksum, verified once the
clock has stopped — which is also what keeps the searches from being elided.
random_device, so two separately built A/B arms shuffleddifferently. Fixed, and
-DPSI_VM_BENCH_SEEDvaries it deliberately.nanoseconds. Fractional now.
leaf and per inner node, the byte limit, and whether each level came out
linear or binary. Without that, two builds' numbers are not comparable, and a
flag that silently failed to take reads exactly like a refuted hypothesis.
What it already says
At the default 512-byte geometry with
intkeys a leaf holds 123 values(492 B) and an inner node 61 (244 B) — so with the 2048-byte limit both
levels take the linear scan, not just the leaf. A descent is therefore several
linear scans over 61 plus one over 123, against
absl::btree_set's binarysearch over 256-byte nodes. That the inner levels are linear too was not
previously visible from any output this benchmark produced.