Reach the fast Parquet codec into the cache, and prune before opening the source

[philcunliffe]

Follow-up on the five deferred findings from the review of PR #1048. Three warranted a change; two did not.

(a) The codec speedup now reaches cache writes

src/core/cache/iceberg/stream_append.js built its ParquetWriter with a codec but no compressors, so hysnappy reached only the sink export encoder and the cache kept hyparquet-writer's own JS snappy.

A correction to the issue's framing first, because it changes what the fix covers: openStreamingAppend has exactly one caller, maintenance.js's compaction sink. Ingest FLUSH goes partition.js -> appendRowsToTable -> icebird's writeParquet, which accepts no compressors at all and cannot be wired without an upstream icebird change. So this reaches the compaction rewrite path, not ingest. That is still the largest single producer of parquet bytes in the process (a rewrite re-encodes every live row of a table), so it is worth having, but "cache ingest" over-states it.

The shared WASM instance moves out of the format-parquet plugin into src/core/util/parquet_snappy.js, so both write paths use one instance. Two would be two WASM memory floors, each sized to its own worst page, for no gain over one sized to the worse of the two. Instantiation is lazy so importing the module does not build a WASM instance in a process that never writes parquet.

Evidence, to the standard #1048's own codec change met. The blast-radius worry was silent corruption at write time, surfacing only on read, so each leg is proven rather than argued:

• Aliasing. hysnappy returns byteArray.slice(outputStart, ...) - a copy out of WASM memory, not a view into it - so the writer may hold a page across later calls. This mattered here and not for the sink encoder: the cache's writer is a custom AbortableWriter that buffers across row groups.
• Codec scoping.ParquetWriter merges { SNAPPY: <js fallback>, ...compressors }, so only SNAPPY is displaced. An UNCOMPRESSED table (write.parquet.compression-codec) is untouched, as is the codec: undefined default.
• Round-trip both ways. Files written through the cache path read back correctly under hyparquet 1.29.2 and under 1.28.2 (the version a user downgrading off Use hysnappy for faster Parquet writes #1048 would land on), through hyparquet's built-in snappy and through hyparquet-compressors, over all-null pages, one-byte values, incompressible noise, highly compressible text, and values fat enough to push pages past the writer's 1 MiB default. The 1.28.2 leg used a scratch install outside the worktree.
• Framing. Raw snappy blocks, not the framed stream format: an independent raw-block decompressor reads them, which the framed format would fail.

Measured, with the honest numbers. The codec itself is much faster: on a 1 MiB textual page, 1.35 ms vs 5.69 ms (4.2x); on a 1 MiB random page, 0.14 ms vs 1.48 ms, and 59 KB out vs 209 KB. End to end it is smaller than that suggests, because compression is a modest share of a compaction's work. A 40,000-row text-heavy compaction, 6 runs each, medians: 2,971 ms vs 3,058 ms (~3% faster) and 9,151,404 vs 9,227,960 bytes (~0.8% smaller). Directionally consistent with the microbenchmark (~140 ms of compression saved on ~32 MB of pages). Real, not dramatic.

(b) and (c) The indexed grep tier gets its fast path back

A footer-free restructuring does exist, on hypgrep's public API. queryIndex is exported and reads only the sidecar; parquetFind does not touch the source until after its own internal queryIndex has pruned. What forced the ordering was purely the call site: columns has to be a resolved array at call time, and computing it needs the source footer.

So queryIndex now runs first, and the source is opened - and its footer read for the physical projection - only once a candidate block survives. A fully-pruned file previously paid a stat plus a 512 KiB footer slice (parquetMetadataAsync's default initial fetch) to prove it had nothing, and pruning to nothing is the common case for exactly the selective query this tier exists to make fast. On a cache of 128 MiB compacted files, a query that prunes everything read ~512 KiB per file where it previously read zero.

That closes (c) at the same time: the source is not opened at all for pruned files, so a compaction or purge that unlinks a data file mid-walk can no longer fail a query that never needed to read it. The ENOENT window narrows back to where it was before #1048.

The cost, stated at the helper: parquetFind re-runs queryIndex and takes no way to be handed the result, so a file that does have candidates decodes its posting bitsets twice. That is CPU over a buffer io.reader already made resident, with no second read, and the index footer is parsed once either way because the metadata rides back into parquetFind. Only a definite "no blocks" shortcuts; every other outcome, failures included, falls through to the existing path, so a poisoned or unreadable sidecar degrades exactly where it did before with the same warning.

No LLP is minted: this restores the IO story LLP 0303 #memory-bound and 0304 #indexed-tier-residency already describe, rather than settling anything new. Both are Accepted and neither is edited.

(d) hyparquet is deduped again, so LLP 0222's claim is true rather than corrected

Re-derived against current master with a real npm install (this repo checks in no lockfile):

+-- hyparquet-writer@0.16.8 -> hyparquet@1.29.2 deduped
+-- hyparquet@1.29.2
+-- hypgrep@0.5.1 -> hyparquet@1.29.2 deduped
+-- hypvector@0.2.2 -> hyparquet@1.26.2 (+ its writer -> 1.26.1)
`-- icebird@0.8.26 -> hyparquet@1.29.1

The drift predates #1048: icebird 0.8.25 (#982, 2026-08-21) began declaring its own exact hyparquet, npm nested a second copy, and nothing reddened because both copies sit above the 1.28.2 floor and only a below-floor copy changes an answer. LLP 0222 #hyparquet-floor's "resolving to a single deduped copy shared with icebird" was true when written and stopped being true silently.

The honest fix turned out to be the dependency change rather than the doc note, because it is nearly free: diff -rq between the hyparquet 1.29.1 and 1.29.2 tarballs shows package.json as the only differing file (it added default export conditions; devDependency bumps). The override moves icebird between two byte-identical src/ trees. So an icebird: { "hyparquet": "1.29.2" } override restores the property the LLP records, and no Accepted doc is edited - its claim is true again.

Two tests hold it, deliberately separate from the existing floor tests, because dedupe is hygiene and the floor is correctness: a dedupe failure must not read as a wrong-rows failure. The comment states the direction of the remedy - if a dependency ever declares a hyparquet above the root pin, move the root pin up, never hold the dependency down onto an older reader.

hypvector's nested 1.26.x copies are out of scope, as hyparquet-floor-pin.test.js already documents: it is an optionalDependency, write-side/vector-side, and does not run icebird's converter.

(e) No change

physicalProjection's columns: [] dependency is documented at the helper and pinned at the library seam by test/core/cache-iceberg-schema-evolution.test.js, "a projection narrowed to nothing still reads one row object per physical row". That test asserts the behaviour directly against hyparquet rather than through its effect, and names itself on failure, so a bump that reinterprets an empty projection reddens by name instead of surfacing as a purge that reports success and spares rows. The pin is sufficient; nothing further is owed unless upstream documents or changes the behaviour.

Tests

Each fails on origin/master and passes here, proven by reverting the source change and re-running:

Test	Pre-fix failure
cache-write-codec.test.js: "the cache compaction writer compresses pages with hysnappy, not the writer fallback"	page of 1600 plaintext bytes was not hysnappy's encoding (stored 1605, hysnappy 1603, fallback 1605)
cache-write-codec.test.js: "every page shape the cache writes reads back through the query path decompressor"	passes on both (it is the round-trip half, not the wiring half)
search-grep-service.test.js: "a file the index prunes to nothing is answered without opening it"	ENOENT: no such file or directory, stat '.../data/....parquet'
hyparquet-floor-pin.test.js: "icebird is held at the root hyparquet pin, not left to nest its own"	the override is absent
hyparquet-floor-pin.test.js: "no root dependency nests a hyparquet of its own"	icebird/node_modules/hyparquet is hyparquet@1.29.1, beside the root 1.29.2

The first test's divergent counter is what gives it teeth: the two snappy implementations agree byte for byte on short and repetitive pages, so without proof that at least one page distinguishes them the assertion would pass on either wiring.

Green: npm test (5,397 pass / 0 fail), npm run typecheck, npm run build:types, npm pack --dry-run, and the smokes query_grep_roundtrip, cache_lifecycle_maintenance, cache_roundtrip, local_parquet_export, purge_removes_cached_rows, incremental_sink_compaction, gateway_claude_capture.

Fixes#1057

[philcunliffe]

Verdict: findings (1 low, fixed) — the correctness core holds up under independent re-derivation

Reviewed at a1e0d333 in a detached worktree with a real npm install from the worktree root (no symlinked node_modules; git status clean before and after, package.json untouched by the install). Everything below about the dependency tree, the round-trip matrix, and the hysnappy internals depends on that install; the code reading does not.

I did not take the PR's evidence matrix on trust. Re-derived independently:

The WASM aliasing question — confirmed, twice, two ways

The installed hysnappy is 1.1.1 (node_modules/hysnappy/js/compress.js), and the returned page is a copy:

constbyteArray=newUint8Array(memory.buffer)// re-derived AFTER memory.grow...returnbyteArray.slice(outputStart,outputStart+compressedSize)

Two things matter and both hold. slice copies rather than views, so the buffering AbortableWriter can hold a page indefinitely. And byteArray is rebuilt after the memory.grow in the same call, so a growth cannot hand back a detached view.

Empirically, against the installed module: held a returned page, then ran three further compressions including a 4 MiB input that forces memory.grow and a 200 KiB input that overwrites the output region. The held bytes were unchanged and still decompressed to the original. held.buffer.byteLength === held.byteLength and byteOffset === 0, i.e. its own ArrayBuffer, not a window into WASM memory. The claim is true.

I also checked the cross-call hash table, since that is the other way a shared instance could corrupt. The WASM stores match candidates as offsets from a fixed input base and then re-verifies each with a 32-bit load against the current input plus a p - match <= 0xffff distance check, so a stale entry can only ever produce a genuine backreference or be rejected. Fuzzed it: 93 inputs (0 bytes to 2 MiB, incompressible / repetitive / all-zero, compressed through the shared instance in scrambled order with every result held to the end) — 0 round-trip failures, and the shared instance's output was byte-identical to a cold instance's on all 93. So there is no cross-call state that changes an answer, and the new test's byte-identity assertion is not fragile for that reason (20/20 stable runs).

Interleaving: the sharpest case is two writers alive at once through the one instance. Opened two openStreamingAppend sinks and wrote 4,000 rows to each alternately, row by row, with targetFileBytes small enough to force file rotation (2 and 4 files) and multiple row groups. Both read back with zero mismatches.

Also probed for a size ceiling: 1 / 8 / 64 / 200 MiB pages and a 120 MiB repetitive page all compress and round-trip.

Round-trip matrix I actually ran

Written through openStreamingAppend (the real path, not a synthetic writer), read back 9 shapes x 3 hyparquet readers x 2 decompressor wirings = 54 cases, all green:

	1.29.2 (current)	1.29.1 (icebird's pre-override)	1.28.2 (LLP 0222 floor, the downgrade target)
built-in snappy	ok	ok	ok
hyparquet-compressors	ok	ok	ok

Shapes: empty table (0 files), all-null, one-byte values, incompressible noise, highly compressible, pages pushed past 1 MiB, a single 3 MB cell, empty strings, multi-byte unicode.

Framing is raw blocks, verified directly rather than inferred: the output carries no ff 06 00 00 73 4e 61 50 70 59 stream header, and its leading varint decodes to exactly the uncompressed length (1200 for a 1200-byte input). Parquet requires raw; this is raw.

compressors merge order

hyparquet-writer/src/parquet-writer.js:35 is this.compressors = { SNAPPY: snappyCompress, ...compressors }, and datapage.js:77 is compressors[codec]?.(pageBytes) ?? pageBytes. Exercised all four cases:

• codec: 'SNAPPY' and codec: undefined -> SNAPPY pages, identical bytes (the writer defaults to SNAPPY).
• codec: 'UNCOMPRESSED' -> UNCOMPRESSED pages, 31,068 bytes vs 2,581; the supplied map is not consulted. 'UNCOMPRESSED' in pq.compressors === false.
• A caller passing its own SNAPPYwins by identity (pq.compressors.SNAPPY === callerFn, and it is the one actually invoked).
• A caller passing GZIP keeps it alongside.

icebird's resolveParquetCodec can only yield undefined | 'SNAPPY' | 'UNCOMPRESSED', so those four cases are the whole space. Both ParquetWriter construction sites in the repo are wired; there are no others.

The grep restructuring

queryIndex is genuinely sidecar-only (hypgrep/src/queryIndex.js: parquetMetadataAsync(indexFile) then n-gram work, no source touch), and parquetFind really does accept and forward indexMetadata into its own queryIndex, so the second footer parse is avoided and nothing is conflated with sourceMetadata. hyparquet's defaultInitialFetchSize = 1 << 19 confirms the 512 KiB figure in the comment, and asyncBufferFromFile is a stat plus per-slice streams, so the saving for a pruned file is exactly the stat plus the footer slice the comment claims, with no handle lifecycle to get wrong.

Equivalence checked rather than assumed: parquetFind shortcuts on the identical blocks.length === 0, so pruning first cannot change an answer, only decline to read. pruned?.blocks.length === 0 short-circuits safely when queryIndex returns undefined (and compileMatcher refuses an empty query, so hypQuery is never falsy anyway); a blocks-less object would throw into the catch and fall through. Aborts re-throw; every other failure falls through to the unchanged path. indexedFiles cannot double-count (the prune branch returns).

The four new assertions have teeth. I copied all three test files onto a fresh origin/master worktree with its own install and ran them: 4 fail / 27 pass, failing exactly on the hysnappy wiring, both dedupe assertions, and the ENOENT prune test. On this branch: 5397 pass / 0 fail / 1 skip.

The dependency override

Re-derived the tarball comparison: diff -rq of hyparquet-1.29.1.tgz and hyparquet-1.29.2.tgz reports package/package.json as the only differing file, and that diff is the version string, three added default export conditions, and two devDependency bumps. No src/ delta, no runtime dependency change. The override moves icebird between byte-identical trees.

Confirmed it pulls nothing else. Full npm ls --all diff, master install vs this branch, is one line: icebird -> hyparquet 1.29.1 becomes 1.29.2 deduped. Nothing added, nothing removed. npm pack --dry-run differs only by the new source file and its two generated type artifacts (1008 -> 1011 files); no packaging or files change.

Green here: npm test, npm run typecheck, npm run build:types, npm pack --dry-run, and the smokes query_grep_roundtrip, cache_lifecycle_maintenance, cache_roundtrip, local_parquet_export, purge_removes_cached_rows, incremental_sink_compaction.

Conventions and LLP

No semicolons (the two ; matches in the touched files are prose inside comments), no U+2014 anywhere, no @typedef, no inline import('...') types, FileMetaData already in the file's @import block, and the one new type-import specifier (cache-write-codec.test.js) is root-anchored .js. Zero files under llp/ in the diff, confirmed. The @refs hold: LLP 0222 has {#hyparquet-floor} and its "single deduped copy shared with icebird" sentence is the thing the override makes true again; 0303 #memory-bound and 0304 #indexed-tier-residency still describe the source correctly, and pruning first strengthens rather than contradicts them.

---

Finding

1. Low — src/core/util/parquet_snappy.js:6 overstated its own reach. The module header read "shared by every parquet WRITE path in this repo". It is not, and this PR's body is the thing that says so: anything routed through icebird's writeParquet takes a codec but no compressors at all. That covers cache ingest FLUSH (src/core/cache/partition.js:151 and :192 -> appendRowsToTable) and, less obviously, stream_append.js's own legacyAppend fallback at src/core/cache/iceberg/stream_append.js:516 for a table it cannot stream into. Separately the grep sidecar build (src/core/search/index_worker_thread.js) is hypgrep's createIndex, which constructs its own ParquetWriter internally with no compressors.

This matters because that header is the canonical explanation for why the compressor is a singleton and where the speedup lands. The PR body goes out of its way to correct the issue's "cache ingest" framing to "compaction rewrite", and then the code comment quietly re-introduces a stronger version of the same over-claim, in the one place a future reader will look. Fixed in d675ade6: the header now names the two paths it serves, names the ones it deliberately does not and why none of them can be wired without an upstream change, and states plainly that the codec reaching "the cache" is the streamed compaction rewrite rather than every byte the cache writes. Comment only, no behaviour change; re-ran the full suite, typecheck, build:types and npm pack --dry-run after it.

No correctness findings. The codec swap, the shared instance, the merge scoping, the prune-first restructuring, and the override each survive independent re-derivation.

New head: d675ade6.

[philcunliffe]

Verdict: clean — no findings. Round 1's Low is fixed, and the failure paths round 1 did not press hold up under injected faults.

Reviewed d675ade6 in a detached worktree with a real npm installfrom the worktree root only. git status was empty before and after the install and is empty now; package.json is untouched by it. Everything below about the dependency tree, hysnappy internals, and the injected-failure probes depends on that install; the code reading does not.

Round 1 was thorough, so I treated its conclusions as claims and spent the effort where it did not look: fault injection into the write path, the fallback writer, module-load behaviour, and grep concurrency.

Aliasing — spot-checked, holds

Not the full matrix, one targeted probe against the installed hysnappy 1.1.1: held a returned page, then compressed a 6 MiB input (forces memory.grow) and a 300 KB input (overwrites the output region). Held bytes unchanged, held.buffer.byteLength === held.byteLength && byteOffset === 0 (its own ArrayBuffer, not a window into WASM memory), and it still decompressed to the original. Confirms round 1's reading of byteArray.slice(...) re-derived after the grow. Moving on.

Also re-checked the small end, where a length-prefix bug would corrupt silently: 0, 1, 2, 3, 4, 63, 64, 65-byte pages all round-trip, and all but the 1000-byte one are byte-identical to the writer's JS snappy.

Failure and partial-write paths — patched hysnappy to throw on the Nth page

The question that matters for a cache write path is whether a mid-stream codec failure can land a corrupt or half-written data file. It cannot, for two independent reasons, both verified rather than argued.

Nothing is written in place.localWriter (src/core/cache/iceberg/resolver.js:167) writes to a .tmp.<pid>.<ms>.<rand> sibling and only renameSyncs onto the final name inside finish(). A file that never finishes never exists under its real name.

Injected throw mid-write. Second append into a live table, compressor throwing from page 3 of 3 batches, with maintenance's own finally (src/core/cache/maintenance.js:1430, if (!streamed && sink.current) await sink.current.abort()):

• throws with the injected error, does not hang or swallow
• zero leftover .tmp.*
• data dir file count unchanged (6 before, 6 after)
• table still readable, exactly the 500 committed rows, ids intact

Injected throw after files had already rolled (targetFileBytes: 1024, failure on page 7): three files had already been finish()ed and renamed into data/ before the throw. They stay on disk as orphans — but listLiveDataFiles returns 1, scanRowsFromTable returns exactly the 200 committed rows, and no .tmp.* remains. Reads go through the manifest, not a directory listing, so an aborted rewrite cannot leak phantom rows into an answer; the orphans are the generation directory the sweep reclaims, which is what commitDataFiles's header already says. That behaviour predates this PR; the codec is simply one more thing that can throw inside it, and it lands in the same place.

Instantiation failure.snappyPageCompressors() is called inside openFileFor, afteropenWriter(dataPath) — but localWriter opens no fd and creates no temp file until the first flush()/finish() (openTmp is lazy), so a throw there leaks nothing at all, and the ParquetWriter constructor (which writes PAR1) never runs. On the encoder side it throws out of resolveEncodeSettings. Neither is CLI boot; see below.

legacyAppend gets no compressors, deliberately, and mixing is safe

Confirmed it is not an oversight: legacyAppend delegates to appendRowsToTable -> icebird's writeParquet, which accepts no compressors parameter at all, so there is nothing to pass. The interesting question is the one round 1 left open — whether a single table holding files from both writers reads back. It does, and this is not hypothetical: ingest FLUSH always takes the icebird path, so every compacted table has files from both.

Built one deliberately: legacy append (300 noise rows) -> streaming append (300) -> legacy again (300), same table. All three live files declare SNAPPY; scanRowsFromTable returns 900 rows, 0 mismatches on the high-entropy column. Nothing assumes a common producer, because nothing reads the producer: the decompressor is selected per column chunk from the file's own codec field.

Codec scoping re-derived at the table level, not just the writer's merge order. Flipped a live table's write.parquet.compression-codec to UNCOMPRESSED and ran a streaming append into it: the new file declares UNCOMPRESSED with total_compressed_size === total_uncompressed_size (1225/1225), while the pre-flip file stays SNAPPY, and all 300 rows read back with 0 mismatches. Supplying compressors does not displace a table's chosen codec — which is the one wiring mistake that would corrupt silently at write.

Module-load cost and the memory floor — the shared move lowered the floor

Instrumented WebAssembly.Instance and counted.

• Importing src/core/util/parquet_snappy.js: 0 instances. First snappyPageCompressors(): 1. Second call: still 1, same function identity. Lazy and singleton, as documented.
• Importing stream_append.jsandformat-parquet/src/index.js together: 0 write-side instances. Running a full streaming compaction: 1. Then invoking the sink encoder's compressor: still 1. Both write paths share one instance — the property the whole module exists for, measured end to end.
• Not doubled, and strictly better than master: master's format-parquet built its instance eagerly at module import (const SNAPPY_COMPRESSOR = snappyCompressor()), so a process that activates the plugin and never writes parquet paid a WASM instance and a WASM-unavailable failure surfaced at plugin activation. Now neither happens until the first page is compressed.
• One instance is created at import time, and it is not this PR's: hyparquet-compressors eagerly instantiates its own nested hysnappy decompressor at module load (hyparquet-compressors/src/compressors.js:9). Read path, pre-existing, unchanged here. Worth recording so a future memory measurement is not misattributed to this change.

Grep restructuring under concurrency

The prune call is argument-for-argument the call parquetFind makes internally — queryIndex({ query, indexFile, indexMetadata }) at hypgrep/src/parquetFind.js:34 and at grep_service.js:333, with parquetFind shortcutting on the same blocks.length === 0. There is no option that reaches the internal call and not ours, so a false prune is not reachable.

Ran the races:

• Three concurrent greps over the same indexed files (two matching, one pruning): identical hit sets, indexedFiles: 2 on all, no cross-talk. The only shared state is a per-call io.reader buffer and hypgrep's own per-generator slice cache.
• Eight concurrent greps: all 2 hits, all agreeing on tier counts.
• Grep racing an unlink of a live data file: the pruning query completes normally (indexedFiles: 2, no hits) with the source gone, which is the improvement this PR claims. A query that genuinely needs the source still throws ENOENT — loud, not silent, and unchanged from before; the PR narrows the window rather than closing it, which is what its body says.

No side effect of searchFile is skipped by the early return: the walk's day-descending break and the tier counters read only hits and file.day.

Conventions, packaging, LLP

No U+2014 anywhere in the changed files. No line-ending semicolons in the diff (the two ; in these files are prose inside pre-existing comments). No @typedef, no inline import('...')type imports (the one await import('hyparquet-writer') at stream_append.js:623 is a pre-existing dynamic value import). Type-import specifiers root-anchored .js. npm run typecheck clean; npm pack --dry-run carries src/core/util/parquet_snappy.js plus its two generated type artifacts, so the published CLI can resolve it.

Zero files under llp/ in the diff. The one @ref this PR edits (0222#hyparquet-floor in hyparquet-floor-pin.test.js:42) holds: the anchor exists and line 80's "resolving to a single deduped copy shared with icebird" is exactly what the override makes true again. The 0209 refs in stream_append.js resolve via <a id="..."> inline anchors, not headings, and still describe the code. 0303 #memory-bound / 0304 #indexed-tier-residency are strengthened by pruning first, not contradicted.

Dedupe re-derived in this install: npm ls hyparquet --all shows icebird's copy deduped at 1.29.2; only hypvector's nested 1.26.x pair remains, which hyparquet-floor-pin.test.js already documents as out of scope. The new test pins overrides.icebird.hyparquet === ROOT_PINS.hyparquet, so bumping the root pin without the override goes red — the directional remedy the comment states is enforced, not just described.

Green here

npm test 5,397 pass / 0 fail / 1 skip. npm run typecheck. npm pack --dry-run. Smokes query_grep_roundtrip, cache_lifecycle_maintenance, cache_roundtrip, local_parquet_export, incremental_sink_compaction, purge_removes_cached_rows all ok.

---

Findings

None. Nothing pushed; head stays d675ade6.

[philcunliffe]

Ship risk: medium

Who could be affected: People whose recorded history has grown enough that HypAware tidies it in the background, and anyone searching that history.

What could happen:

• Tidying rewrites stored history in place. If the new, faster way of shrinking that data were wrong, history could come back garbled later, with the old copy already gone.
• Searches could quietly return fewer results, since a search now skips files an index says cannot match before opening them.
• Everyone installing HypAware resolves a slightly different set of supporting libraries.

Why this level: The affected area is a person's own local history and search, which cannot be undone once rewritten. It does not touch sign-in, privacy, or anything leaving the machine.

What was checked: History was written and read back through the real code with 6,000 records of every awkward shape, plus 490 shrink-and-restore checks, all exact. Searches gave identical results to today's version on 62 tries. The full test suite (5,397 checks) and six workflow checks passed, and the library change was confirmed to be one duplicate removal and nothing else.