feat(search): rank with BM25 over a real lexical index

[drewstone]

Why

searchKnowledge ranked with rankByTokens/tokenScore: substring includes against hand-picked weights (200/50/20/5/3/1), no inverse document frequency, no length normalization, no term-frequency saturation. On a store of ~1,100 claim pages a term that appears in almost every page counted as much as a rare one, and a long page outranked a short one by repetition. Question dedup and kb_search in the consumer both run through this path.

What

BM25 replaces the hand-weighted scorer. Nothing selects the old one; it is deleted.

• New pure src/lexical-index.ts (no dependency, no native module, importable at the edge): buildKnowledgeLexicalIndex(pages, { tokenize, fieldBoosts }) builds an inverted index with field-boosted term frequencies (title 3, path 2, body 1), document lengths, average document length, and document count. scoreBm25(index, tokens, { k1 = 1.2, b = 0.75 }) scores with the Lucene IDF ln(1 + (N - df + 0.5) / (df + 0.5)), ordered by score then path.
• tokenizeText is the token stream; tokenizeQuery is its distinct-token form and moves into the same module, so indexing and querying share one tokenizer and the two vocabularies cannot drift. The split pattern, CJK bigram expansion, and stop list are carried over byte-identically.
• searchKnowledge keeps its signature, hit shape, normalizedScore, snippet, reasons, RRF-with-graph fusion, and path tie-break. The lexical list is BM25 ordered inside four phrase tiers (exact title or path > title contains > body contains > bag of words), so an exact lookup cannot be overtaken by term repetition.
• KNOWLEDGE_SEARCH_RETRIEVER_ID = 'bm25-rrf-v1' is exported and used in the README and receipts-doc examples, so one identity is declared in one place instead of each caller inventing a string.
• FileSystemSearchProvider builds the lexical index once per page index and drops both together on refresh/invalidate(). SearchKnowledgeOptions.lexicalIndex accepts a prebuilt index for repeated queries and refuses one built from other pages.
• CLI help, README search paragraph, docs/knowledge-use-receipts.md example, CHANGELOG (10.1.0, additive).

Simplification

Simplification: rankByTokens + tokenScore + the search.ts STOP_WORDS/tokenizer copy deleted; one tokenizer (tokenizeText/tokenizeQuery) now owns indexing and querying; one exported retriever identity replaces two hand-written doc strings.
Net: +240 / -74 lines across 12 files; 1 scorer, 1 duplicate tokenizer seam and 2 invented retriever ids removed; no flag, no second scorer.
Tests: +8 (IDF discount, length normalization, tf saturation, index/query tokenizer drift incl. CJK, exact-title tier survives a term-heavy page, determinism under page reorder, near-duplicate cluster recall over unrelated prose, filtered search against a whole-corpus index + refusal of a foreign one), -0 deleted, considered and verified: `git grep rankByTokens|tokenScore origin/main -- '*.test.ts' tests/` returns nothing, because both were private to search.ts; the only test importing search on main is src/search.test.ts, whose five tests (overload, kind/tag filters, predicate, empty filters, limit refusal) are behavior-level on searchKnowledge, survive verbatim, and pass.
Not done here: benchmarks/adapters.ts keeps its own tokenize/tokenOverlap — a different stop list and split for the memory-benchmark adapter, coincidental shape rather than shared intent; rag-eval/near-duplicates.ts keeps its shingle index (different purpose, per #148).

Rule 1 — absence proven before adding

grep -rn "bm25|BM25|idf|IDF|inverted index|avgdl|lexical" src tests on origin/main (8a54446) returns no ranking implementation: only a comment in claim-ledger.ts, benchmarks/adapters.tstokenOverlap (memory-adapter scoring), and the shingle index inside rag-eval/near-duplicates.ts. No BM25 in agent-eval or agent-sdk either. Nothing existed to extend.

Proof (local, macOS, node 24.11.1)

• pnpm run typecheck (src + contracts): clean. pnpm run lint: 231 files, no findings.
• Focused suites — src/lexical-index.test.ts, src/search.test.ts, tests/filesystem-search-provider.test.ts, tests/retrieval-eval.test.ts, tests/core.test.ts, tests/rag-eval.test.ts: 66 passed, 0 failed. Existing search/provider/retrieval-eval assertions pass unchanged.
• Tests kept, each naming the failure it prevents: a corpus-wide term no longer outranks a rare one (IDF); a long page no longer beats a short one at equal term frequency; 50 repeats stay under 3x one occurrence (saturation); the query vocabulary equals the indexed vocabulary, CJK bigrams included (tokenizer drift); an exact title match stays ahead of a page repeating the query terms; reordering the pages returns identical ids, scores, and ranks; the near-duplicate cluster (the feat(quality): measure and gate near-duplicate knowledge pages #148 fixture text) is recalled ahead of unrelated prose; a filtered search scored against a whole-corpus lexical index matches the uncached path and a foreign index is refused.
• pnpm run build, pnpm run api:surface (10 additive exports, none removed), pnpm run check:version-bump: 10.0.0 -> 10.1.0 (minor, additive), rebased onto the receipt-reference release (feat(receipts): reference the visibility snapshot from a retrieval receipt #163).
• Focused suites after the test triage, re-run on the rebased head — src/lexical-index.test.ts, src/search.test.ts, tests/filesystem-search-provider.test.ts, tests/retrieval-eval.test.ts, tests/core.test.ts, tests/rag-eval.test.ts, src/knowledge-use-receipts.test.ts: 78 passed, 0 failed.
• Full pnpm test: 690 passed, 16 skipped, 73 failed. Every failure is environmental on this machine, not from this change: 48 in tests/kb-improvement/* (exact knowledge candidate workflows require Linux directory descriptors, identical on a clean origin/main checkout) and 25 in tests/version-bump-check.test.ts, which times out at 15s under parallel load and passes in this same worktree with --testTimeout 90000 and on a clean origin/main checkout. CI (ubuntu) runs all of them.

Closes#132

[tangletools]

✅ Auto-approved drewstone PR — 2d51fe90

This PR was opened by the trusted drewstone account.

This approval is provisional and was applied by the local stand-in because the pr-reviewer webhook host is unreachable (2026-08-21). CI on this head is fully green. The full PR reviewer audit re-runs via the resweep when the service returns and will publish findings if it detects issues.

[tangletools]

✅ Auto-approved drewstone PR — 45fac171

This PR was opened by the trusted drewstone account.

This approval is provisional and was applied by the local stand-in because the pr-reviewer webhook host is unreachable (2026-08-21). CI on this head is fully green. The full PR reviewer audit re-runs via the resweep when the service returns and will publish findings if it detects issues.