Extraction reads only the open graph

[WaylandYang]

Stacked on #735. Decision 0044, decision 2: extraction writes only the open graph. There is no released version and the user decided there is no legacy layer, so the typed extraction path goes, and with it the per-base switch that selected it. Alignment (cut 2) is now the only way typed facts come from documents.

Removed (19 files, +116 / −6449)

• crates/utopia-server/src/extraction.rs (4664 → 753 lines): the typed body of run, the ontology prompt lists and per-chunk retrieval, predicate matching, span verification, the name-shape guards (is_entity_name, clause_suspect, the word lists), direction correction, attribute and qualifier writing, and the post-extraction signature sweep and type-resolution / ontology-bootstrap enqueues. What stays is the job entry and what the open path shares: rate-limit retries, the confidence ceiling for described chunks, drop signals, the incomplete-run rule, resolve_handle and namesake reviews. The two connected tests around namesake handles were rewritten around resolve_handle.
• crates/utopia-server/src/ontology_index.rs: the extraction-side ontology gate (gate_required). Ontology embedding and nearest-neighbour suggestions stay.
• crates/utopia-extract/src/lib.rs (2222 → 1029) and normalize.rs (deleted): the typed contract (Extraction, ExtractedFact, the ontology prompt, parse_response) and its normalisation. Kept: the open contract, KnownEntity, JSON block repair, the adjudication messages, the quantity and time parsers used elsewhere. The unicode-segmentation dependency is gone.
• knowledge_bases.open_extraction (migration 0063 drops the column) and its plumbing in the model, kbs::update, the API request, the web type and the bench.
• 28 drop-reason codes nothing emits any more, with their web copy.

Not removed on purpose: predicate_match (used by ontology bootstrap), ontology bootstrap and type resolution jobs (nothing enqueues them after extraction any more; they remain callable from the ontology page), the ontology-misses panel (no new rows will arrive), proposed_predicates and adoption (typed rows only), signature checks. They idle until cut 2 gives them their input; deleting or reshaping them is that cut's job.

Two findings from a 25-document batch, fixed here

• A description becomes an entity only when a statement points at it. The model lists every noun phrase of a paragraph in e; the batch had 848 entities of which 616 were described and 181 of those carried no statement at all (some 150-character clauses). Described things are now deferred and created on first reference; the per-document dedup stays. Same batch re-extracted: 705 entities (476 described, 50 of them referenced only from a qualifier), no case-variant duplicates among named entities.
• An untyped mention now meets its untyped namesake. resolve_mention matched candidates with type_id = $2, which is never true for NULL; on the open path every entity is untyped, so "Securities and Exchange Commission" and its upper-case spelling became two entities. The query uses IS NOT DISTINCT FROM (the 0009 trap), with a connected test.

Behaviour after this change

Every document, memory logs included, is read by the open path. A fresh base holds open statements (phrase, qualifiers, time words, evidence offsets) and name facts; no typed facts, so conflict and signature queues, timelines and ontology proposals stay empty until alignment lands.

Tests

cargo clippy --workspace -D warnings clean; utopia-extract 31 and utopia-server 276 unit tests pass; connected: the six extraction tests, no_predicate_still_shows, an_open_statement_keeps_the_documents_words, a_pending_statement_keeps_the_documents_words, a_fact_awaits_a_nod, exploration_describes_the_data. End to end on a fresh database (the same flow as #731, scripted): 14/14 checks. A base is created without any extraction switch; a document yields open statements only (14), and every evidence offset and time mention reproduces the text; entity search, entity detail (labelled by phrase), graph overview, RDF export and the MCP entity_facts tool all read them; remember over MCP queues two pending open statements with phrase and quote span, and the nod writes one with its evidence.

🤖 Generated with Claude Code