A statement keeps the document's words

[WaylandYang]

Closes #729. Cut 1 of 0044 (#710): the open graph lands in the ledger and extraction writes only that layer (decision 2). This is the breaking upgrade the record allows: no released version, no legacy layer. The typed extraction path stays as an opt-out per base (open_extraction = false), which is the optional bound pass of decision 3, kept until alignment reaches parity.

Ledger (migration 0061)

• facts.layer ('typed' | 'open') and facts.phrase; an open row keeps the document's relation phrase and has no predicate (facts_open_statement_shape). Dedup is on (base, subject, phrase, object), which also closes the NULL-predicate dedup hole of 0010.
• statement_qualifiers: qualifiers keyed by the document's own role word ("to", "amount", "compared to"), value or entity.
• time_mentions: the words a statement uses for time, with the char offset in the chunk. Never a computed date; interpretation comes with 0045.
• fact_evidence.quote_start/quote_end: server-computed char offsets of the quote in the chunk.
• entities.description: a described, unnamed thing is an entity. It gets no known_as fact and is excluded from name recall, so a description never bridges two documents.
• knowledge_bases.open_extraction (default on), settable through PATCH /kbs/{id}; off selects the typed path.

Both clocks: attested_from is always set; attested_at comes from the document date only when doc_time_source is content or source (#714). Open rows write no valid_*.

Contract

utopia_extract::open: a prompt with no ontology, no document date and no dates to compute; a compact reply of e things (name or description, kind word, named/described), s statements (subject name, phrase, object name or literal value, qualifiers keyed by role word, when/ended time words, verbatim quote) and n other names. Parsing is per item: malformed items are counted, truncated replies are repaired to the last complete item.

Statements name their subject and object in words and carry their own quote. The first version used numeric ids and quote indices to save tokens; measured on a dense earnings-release chunk, the model mixed the ids up ("NVIDIA has reached AI" for "AI has reached its inflection point"). Names fixed it (table below).

Write path

extraction_open::run_open, entered from extraction::run unless the base opted out or the document is a memory document (memory facts await a nod and have no phrase column yet). It reuses the chunk loop, epoch checks, rate-limit retries, the identity path (resolve_handle, name facts of 0041) and the govern/adjudicate enqueues; it skips the ontology lists, predicate matching, the temporal reconcile, signature checks, type resolution and ontology bootstrap. Names are resolved against the reply's list and the document's earlier things; an unlisted object becomes a literal value with a signal (object_undeclared), an unlisted subject is a drop (unknown_ref). Quotes are located in the chunk (quote_not_in_chunk when not verbatim); a time mention is recorded only when its words occur in the statement's own quote (time_not_in_quote), because the model attaches one time to unrelated sentences otherwise.

proposed_predicate = phrase is written on the evidence, so every read path that already tolerates a NULL predicate shows an open statement under its phrase. Adoption (proposed_predicates, adopt) is restricted to typed rows so a phrase is never auto-adopted as a relation type; adjudication profiles and disambiguators read open rows through LEFT JOINs.

Measured (deepseek-v4-flash extraction; deepseek-v4-pro judges)

Reference units of three domains (10 paragraphs each, reference fact lists from the prototype bench). Six rounds: ids contract (r1); names contract (r2); prompt rule 4 no longer unfolds a bare belonging (r3, r4 same prompt); temperature 0 (r5, r6 same prompt). Columns: statements / judged stated / not stated / stated-fact coverage.

round	FDA approvals	CN statistics bulletin	NVDA 8-K and release
r1 ids	131 / 83.2% / 3.1% / 54.2%	104 / 89.4% / 1.0% / 55.5%	71 / 88.7% / 0.0% / 46.8%
r2 names	141 / 94.3% / 2.8% / 60.3%	139 / 98.6% / 0.0% / 48.9%	108 / 93.5% / 0.0% / 61.7%
r3 rule 4	98 / 91.8% / 1.0% / 48.6%	101 / 97.0% / 0.0% / 56.2%	102 / 93.8% / 0.0% / 56.7%
r4 rule 4	143 / 95.8% / 0.7% / 59.2%	124 / 97.6% / 0.0% / 55.5%	101 / 80.2% / 0.0% / 71.6%
r5 temp 0	142 / 94.4% / 0.7% / 58.1%	128 / 93.8% / 6.3% / 41.6%	99 / 80.8% / 0.0% / 64.5%
r6 temp 0	131 / 93.1% / 1.5% / 52.5%	130 / 92.3% / 0.8% / 58.4%*	96 / 97.9% / 0.0% / 56.7%

• 23 reference facts unjudged (judge endpoint errors); counted as not covered.

What the rounds say:

• The names contract removed the id mix-ups: misworded fell from 10–14% to 2–7% everywhere, and the NVDA misworded rows that remain are financial-table cells whose period column is not captured (a table-reading follow-up, not fabrication).
• Not stated is at or under 2% in 16 of 18 corpus-rounds. The two exceptions: FDA r2 (2.8%: two unfoldings of a bare belonging, fixed by rule 4, and two the judge called strictly) and CN r5 (6.3%: eight rows from one patent table where the model read column membership as "belongs to"; not reproduced in r6).
• Rule 4 did not cost coverage (r3 was a sparse run; r4 with the same prompt is back at r2 levels). Run-to-run density at the endpoint default temperature swung per unit by 3x (31 / 8 / 13 statements on one paragraph); at temperature 0 the same units read 8/9, 12/14, 24/24, 11/11.
• Coverage of stated reference facts sits at 50–60% per corpus with 10-point swings between runs; the misses are figures in dense sentences (the model states the relation and drops the amount) and the year in a bulletin's title that no longer attaches to every statement (document time context, 0045).

NVDA full documents on scripts/bench/recall.mjs (52 truth items, no ontology in the base): 38/52 with ids, then 44, 46, 45 with names (1σ ≈ 2.7 items). The README's typed baseline (90%+) was measured with a schema.org pack and a different model, so it is not a like-for-like number; the typed path on this branch is unchanged.

Tests

• Store (connected): an_open_statement_keeps_the_documents_words (phrase shows in overview, entity detail, neighborhood, paths and export; second observation reuses the row; qualifier, time mention and offsets round-trip; not offered for adoption; shape constraint), a_described_thing_is_an_entity_without_a_name, a_time_mention_is_words_not_a_date. Existing no_predicate_still_shows, a_name_is_a_fact, a_qualifier_is_not_the_edges_identity, adjudication and disambiguator tests pass.
• Contract: 17 unit tests in open.rs. Server: offsets are characters, a time is located inside its own sentence only, names match up to whitespace and case.
• Smoke on a fresh database with the flag on: statements, qualifiers, time mentions and evidence offsets all reproduce the text; entity detail shows the phrase as the label.

Bench

scripts/bench/recall.mjs runs the open path by default and --typed the old one, scoring both with the same rules (qualifiers and time words included), so the two paths are comparable. scripts/bench/judge_open.mjs samples open statements and has a judge model read the chunk: stated / misworded / not_stated, with the 0044 threshold of at most 2% not stated.

Not in this cut

Alignment and typed facts derived from open statements (cut 2), time interpretation and document time context (0045), identity profiles, a layer marker in the UI (open rows render as today's unnamed relation), deleting the typed path. Until cut 2 lands, a freshly extracted base holds open statements only: no typed facts, so the conflict and signature queues, timelines and ontology proposals stay empty for it.

🤖 Generated with Claude Code