A search does not hold a runtime worker, and its two channels run side by side

[WaylandYang]

Closes #515.

What changed

retrieval::hybrid called SearchIndex::search straight from the async function. That call is synchronous Tantivy work (tokenise, walk the mmap'd segments, fetch each hit), so it held a runtime worker for its whole duration; the other six Tantivy call sites in the server already wrap theirs in spawn_blocking, and this was the hottest one left. The two channels also ran in series: BM25, then the settings row, then the embedding round-trip to the model, then the vector query, so a question paid the sum of both legs.

Now:

• BM25 runs on the blocking pool via spawn_blocking, the same shape as the reindex and delete sites.
• The two channels run side by side with tokio::join!. Latency becomes max(bm25, embed + vector), and since the embed leg is a network hop, the BM25 leg is close to free.
• The vector leg is its own function returning Option. No embedding model, a failed embedding request, or an empty answer all yield None and the search continues on BM25, which is the degradation the module header promises. join! rather than try_join! so a model outage stays a degradation and does not become a failed search. A failure of the vector query itself is still an error: that is the database, not the model.
• The list order into rrf_fuse is pinned: BM25 first, vector second, whichever finished first. RRF ignores order today; it would not if the channels were ever weighted.
• search_docs in the chat tools had the same shape on the Charter index and gets the same wrap.

The record-axis asymmetry the header documents (as_of complete on the vector and fetch paths, not on Tantivy) is untouched.

Tests

• rrf_fuse had no tests. Four pure ones in utopia-search: one list keeps its order, two lists follow the reciprocal-rank arithmetic, an empty list contributes nothing, the limit cuts after fusion.
• channel_lists is the pure seam for the ordering rule: the channels keep their places, and a missing vector channel leaves BM25 alone.
• Three database-backed tests in retrieval_tests.rs, with a temp Tantivy index and wiremock standing in for the embedding endpoint: a search without an embedding model still answers; a 500 from the endpoint degrades to BM25 rather than failing; with both channels the chunk both found ranks first and the vector-only chunk still comes back. They use test_db::url() and ran green with UTOPIA_TEST_REQUIRE_DB=1 against a database at dev's migrations.

Not tested: "a search never occupies a runtime worker". On a small index the search takes microseconds and any assertion on it would flake; spawn_blocking is the established pattern here and the change is visible in the code.

Acceptance

Not done in this PR. What it buys shows under load: run twenty concurrent /search requests against a base of a few thousand chunks while polling /health every 100 ms. Before, the health latency follows the searches; after, it does not. Latency p50/p95 needs a base large enough for the BM25 leg to be measurable; the local bases are hundreds of chunks.

🤖 Generated with Claude Code