feat(search): port hybrid RRF + alpha-blend pipeline from semble

[amondnet]

Summary

Port `src/semble/search.py` → `src/search.ts`. Implements the hybrid search pipeline that fuses semantic (dense) and BM25 (sparse) results via Reciprocal Rank Fusion, then applies optional code-tuned reranking.

What's included

• `RRF_K = 60`
• `_rrfScores(scores)` — raw scores → `1 / (RRF_K + rank)` (rank 1 = highest raw score)
• `_searchSemantic(...)` — encodes query, calls `semanticIndex.query`, converts cosine distance → similarity (`1 - distance`)
• `_searchBm25(...)` — tokenizes query, calls `bm25Index.getScores`, excludes zero scores, sorts top-k
• `_sortTopK(arr, topK)` — descending-index helper
• `search(query, model, semanticIndex, bm25Index, chunks, topK, options)` — full pipeline

Pipeline (preserved exactly)

1. `resolveAlpha(query, alpha)` — auto-detect via `isSymbolQuery` (0.3 for symbols, 0.5 for NL)
2. Over-fetch `topK * 5` candidates from each backend
3. RRF-normalize each score set independently
4. Union sorted by `startLine` to counteract hash-iteration nondeterminism
5. Combine: `alpha * normSemantic + (1 - alpha) * normBm25`
6. If `rerank`: `boostMultiChunkFiles` (in-place) → `applyQueryBoost` → `rerankTopK` with `penalisePaths = alpha < 1.0`
7. Else: plain top-k sort

Stubs

Sibling units (types, tokens, ranking/{weighting,boosting,penalties}) are not yet on `main`, so minimal stubs are inlined and marked with `// TODO(integration): replace with import from ...`. The stub `rerankTopK` mirrors Python's file-saturation logic without path penalties; the stub `applyQueryBoost` is identity.

Tests

20 tests across:

• `_sortTopK` (descending order, clamping, empty input)
• `_rrfScores` (rank math, empty input, rank-1 = 1/61)
• `_searchSemantic` (distance → similarity, selector pass-through)
• `_searchBm25` (zero exclusion, empty tokens, selector mask)
• `search()` — alpha extremes (0.0 / 1.0), RRF normalization, sort stability via startLine, empty inputs, rerank toggle, file-saturation decay, auto-alpha for symbol queries

Source of truth

`/Users/lms/.ask/github/github.com/MinishLab/semble/main/src/semble/search.py`

🤖 Generated with Claude Code

---

Summary by cubic

Ports the hybrid search pipeline from Semble to TypeScript as src/search.ts, combining semantic and BM25 results with RRF and alpha blending, plus optional code-aware reranking. Adds tests to validate scoring, blending, and output stability.

• New Features

  • New search() that blends RRF-normalized semantic and BM25 scores; over-fetches (topK * 5) and orders the union by startLine for stable results.
  • Auto alpha (0.3 for symbol-like queries, 0.5 otherwise) with override via options.alpha.
  • Optional rerank adds multi-chunk file boost and file-saturation decay; sets penalisePaths when alpha < 1.0.
  • Helpers and types: _searchSemantic (1 − distance), _searchBm25 (tokenize + selector mask + zero-filter), _sortTopK, _rrfScores with RRF_K = 60, and minimal interfaces (Model, SelectableBasicBackend, Bm25Index). 20 tests cover core math, alpha extremes, stability, rerank, and symbol-query behavior.

• Migration

  • Tokens and ranking modules are stubbed; replace with real imports when those units land.
  • To filter candidates, pass options.selector: Uint32Array; BM25 builds its mask internally.
  • Set { rerank: false } to return plain top‑k by blended score.

^{Written for commit 0c59cc6. Summary will update on new commits.}

[gemini-code-assist]

Code Review

This pull request introduces a hybrid search implementation in TypeScript, combining semantic search and BM25 keyword search using Reciprocal Rank Fusion (RRF), along with code-tuned reranking features and comprehensive unit tests. The review feedback highlights two key areas for improvement: first, correcting the minSelected calculation in rerankTopK to ensure the early-exit optimization works as intended; second, enhancing the candidate sorting logic by incorporating filePath alongside startLine to guarantee fully deterministic search results.

[cubic-dev-ai]

4 issues found across 2 files

Architecture diagram

sequenceDiagram
participant Client as Caller (search.ts)
participant Resolve as resolveAlpha()
participant Token as tokenize() (stub)
participant SemSearch as _searchSemantic()
participant Bm25Search as _searchBm25()
participant Model as Model (embedding)
participant SemanticIdx as semanticIndex (SelectableBasicBackend)
participant Bm25Idx as bm25Index (Bm25Index)
participant RRF as _rrfScores()
participant Sort as _sortTopK()
participant Rerank as rerankTopK (stub)
participant Boost as boostMultiChunkFiles()
participant QueryBoost as applyQueryBoost() (stub)
Client->>Resolve: resolveAlpha(query, alpha)
alt auto-detect
Resolve->>Resolve: check _SYMBOL_QUERY_RE
alt symbol query
Resolve-->>Client: 0.3
else natural language
Resolve-->>Client: 0.5
end
else explicit alpha
Resolve-->>Client: provided alpha
end
Note over Client: Over-fetch: candidateCount = topK * 5
Client->>SemSearch: _searchSemantic(query, model, index, chunks, candidateCount, selector)
SemSearch->>Model: model.encode([query])
Model-->>SemSearch: queryEmbedding (Float32Array[])
SemSearch->>SemanticIdx: semanticIndex.query(vectors, candidateCount, selector)
SemanticIdx-->>SemSearch: [[chunkIndex, cosineDistance], ...]
SemSearch->>SemSearch: convert: score = 1 - distance
SemSearch-->>Client: SearchResult[] (semantic)
Client->>Bm25Search: _searchBm25(query, bm25Index, chunks, candidateCount, selector)
Bm25Search->>Token: tokenize(query)
Token-->>Bm25Search: tokens[] (empty → return [])
Bm25Search->>Bm25Search: selectorToMask(selector, chunks.length)
Bm25Search->>Bm25Idx: bm25Index.getScores(tokens, mask)
Bm25Idx-->>Bm25Search: Float32Array scores
Bm25Search->>Sort: _sortTopK(scores, topK)
Sort-->>Bm25Search: Uint32Array indices (descending)
Bm25Search->>Bm25Search: filter zero/non-positive scores
Bm25Search-->>Client: SearchResult[] (BM25)
Note over Client: Build raw score maps
Client->>Client: new Map<Chunk, number>(semantic results)
Client->>Client: new Map<Chunk, number>(BM25 results)
Note over Client: Apply RRF independently
Client->>RRF: _rrfScores(semanticScores)
RRF->>RRF: sort descending by raw score → rank 1..N
RRF->>RRF: score = 1 / (RRF_K + rank)
RRF-->>Client: Map<Chunk, number> rrfSemantic
Client->>RRF: _rrfScores(bm25Scores)
RRF-->>Client: Map<Chunk, number> rrfBm25
Note over Client: Union chunks sorted by startLine
Client->>Client: collect all chunk keys from both maps
Client->>Client: sort by chunk.startLine (stable ordering)
Note over Client: Combine scores
loop for each unique chunk
Client->>Client: alpha * rrfSemantic + (1 - alpha) * rrfBm25
end
alt rerank = true
Client->>Boost: boostMultiChunkFiles(combinedScores)
Boost->>Boost: per-file sum & best chunk
Boost->>Boost: add boostUnit * (fileSum / maxFileSum) to best chunk
Client->>QueryBoost: applyQueryBoost(combinedScores, query, allChunks)
Note over QueryBoost: stub: identity function
QueryBoost-->>Client: combinedScores (unmodified)
Client->>Rerank: rerankTopK(combinedScores, topK, { penalisePaths })
Note over Rerank: stub: file-saturation decay
Rerank->>Rerank: sort descending, apply decay on file saturation
Rerank-->>Client: [Chunk, number][]
else rerank = false
Client->>Client: plain sort by combined score, topK
end
Client-->>Client: SearchResult[] (final)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[amondnet]

Applied 0, deferred 6 — all six review comments conflict with semble parity or target deliberate stubs already scoped to follow-up units.

Deferred (parity-load-bearing) — all 4 algorithm comments would diverge from the upstream semble implementation we're porting:

• minSelected prune in rerankTopK (gemini + cubic): upstream lets selected grow past topK and recomputes min over the full list; that's the algorithm being ported.
• filePath tiebreaker in candidate sort (gemini + cubic): upstream sorts by start_line only.

Deferred (scoped to follow-up units) — the 2 cubic comments on stub functions:

• applyQueryBoost no-op (cubic): explicit stub, full implementation lands with ranking/boosting.ts (unit-5).
• penalisePaths ignored (cubic): explicit stub, full _file_path_penalty lands with ranking/penalties.ts (unit-6). Flag is wired through search() ready to consume the real penalty function.

Inline replies posted on each of the 6 line comments with detailed rationale citing the upstream Python source. All 20 tests still pass (bun test src/search.test.ts).