Ingest embeds four batches at a time, and both ingests share the loop

[WaylandYang]

Closes #513.

What changed

The ingest pipeline embedded one batch of 16 chunks, waited for it, then sent the next. Nothing in the loop depended on the batch before it: vectors are paired by position within a batch and independent across them. The same loop sat verbatim in memory_ingest. The fix the issue points at was already in ontology_index.rs, with the incident that shaped its ceiling written above it.

• One helper, embed_pending, for both call sites. It fetches the chunks still without a vector, batches them, and runs the batches through buffer_unordered(EMBED_JOBS). Each batch holds one permit on the shared embedding gate only for its own request and write, as before.
• EMBED_JOBS = 4, the same as the ontology backfill, and the reasoning is above the constant: the gate (model_concurrency, default 10) is shared by the backfill, type resolution and every question's query embedding. Ingest is foreground and has the better claim, but it is also the caller that can arrive with a thousand chunks; unbounded, it starves everything else, which is the recorded incident. Two callers at 4 sum to 8 and leave 2 for the rest.
• EMBED_BATCH stays at 16. The ontology side warns that batch size has to be measured on real text; chunk text is much longer than a class label and 16 may well be right for a different reason. Concurrency and batch size are two knobs, and turning both in one change makes the result unreadable.
• A count mismatch abandons the batch whole, and the comment now says why: pairing is positional, so one missing vector misaligns the rest and writes one chunk's vector onto another, where it is never visible again. The previous message was "Embedding 返回数量不匹配" with no reasoning.
• A failed batch fails the document. Serial code got this by unwinding; the concurrent version returns the first error explicitly, the stream is dropped and in-flight batches are cancelled, and process_document marks the document failed with the reason instead of leaving it in embedding. Vectors already written stay; a rerun only embeds what is still missing.

Tests

pipeline_tests.rs, database-backed, with wiremock standing in for the embedding endpoint. The fake derives each vector from the text it was given, so pairing is checked by reading the rows back, and it records every request's arrival time and size.

• every_chunk_gets_the_vector_of_its_own_text: 40 chunks, three batches, four in flight, completing out of order; every stored vector matches its own text. Request sizes are 16, 16 and 8, so the remainder is sent; an empty document makes no request.
• a_batch_that_answers_with_the_wrong_count_is_abandoned_whole: 15 vectors for 16 texts, nothing written.
• the_embedding_gate_is_never_held_beyond_its_ceiling: twelve batches with a 150 ms response delay; the peak number of requests in flight, computed from arrival times, never exceeds EMBED_JOBS, batches do overlap, and the whole run takes less than half of serial.
• a_failed_batch_does_not_strand_the_document: the real process_document path with a 20-paragraph blob; the second request returns 500; the document lands in failed with the reason.
• a_document_is_fully_embedded_before_it_is_ready: the real path succeeds; status is ready, chunk count matches, no chunk is left without a vector.
• a_memory_episode_embeds_by_the_same_path_as_a_document: memory_ingest embeds its chunks with the same pairing.

All ran green with UTOPIA_TEST_REQUIRE_DB=1 against a database at dev's migrations. As with the other server-side database tests, CI's backend job has no database and skips them; they run locally.

Acceptance

Not done here; it needs a real embedding endpoint and a document of several hundred chunks. Time the embedding stage before and after, and run an ontology backfill at the same time to confirm neither starves; that second one is the recorded incident and is the one worth doing carefully.

🤖 Generated with Claude Code