From 824f98964251afdaf79c88f85533d947dc75ac00 Mon Sep 17 00:00:00 2001 From: WaylandYang Date: Tue, 15 Sep 2026 14:18:40 +0800 Subject: [PATCH 1/7] 0044 proposes reading a document in stages Co-Authored-By: Claude Opus 5 Signed-off-by: WaylandYang --- .../0044-a-document-is-read-in-stages.md | 259 ++++++++++++++++++ docs/decisions/README.md | 1 + scripts/bench/audit.sql | 90 ++++++ 3 files changed, 350 insertions(+) create mode 100644 docs/decisions/0044-a-document-is-read-in-stages.md create mode 100644 scripts/bench/audit.sql diff --git a/docs/decisions/0044-a-document-is-read-in-stages.md b/docs/decisions/0044-a-document-is-read-in-stages.md new file mode 100644 index 000000000..de9af58a4 --- /dev/null +++ b/docs/decisions/0044-a-document-is-read-in-stages.md @@ -0,0 +1,259 @@ +# 0044 · A document is read in stages + +- **Status**: Proposed · nothing built · the next step is an offline prototype of stages 0 and 3, measured on five benches against today's extractor · prompt rules for today's extractor are frozen until it reports +- **Written**: 2026-09-15 (conventions in the [README](README.md)) +- **Related**: + - Evidence that motivated this record: + - [0006](0006-ontology-scale-and-the-prompt.md): the ontology slice in the prompt. + - [0039](0039-a-chunk-is-what-extraction-sees.md): chunks as the unit of extraction. + - [0041](0041-a-name-is-a-claim-about-an-entity.md): names as facts and the identity bench. Its cuts 2–4 fold into decision 7. + - [0022](0022-an-unknown-date-is-not-an-open-one.md): the temporal engine this record keeps. + - #701: the per-chunk ontology budget. + - #705, closed: sentence citations; names carry recovery. + - #709: cross-script name vectors. + - Records this one extends: + - [0025](0025-governance-reads-the-ledger-before-it-decides.md) and [0043](0043-every-review-queue-is-governed.md): governance of every queue. Decision 8 changes what the queues hold. + - [0003](0003-ontology-growth-loop.md): the ontology grows out of the corpus. Decision 5 is where it grows from. + +> A lease signed in 2016 is amended eleven times. Each amendment cites the lease and its earlier amendments by title and date. Today every 1,000-character chunk of every amendment asks one model call for all of the following at once: +> +> - the entities, their names, aliases and types, chosen from 200 candidate classes; +> - the facts, their predicates chosen from 180 candidate relations, and their direction; +> - values, units, validity dates, quotes and the words that name each side. +> +> The chunk cannot see the rest of the document. "The Fifth Amendment dated February 18, 2020" comes back as its own entity, with its date inside its name. A cited list of amendments comes back as "the Ninth Amendment is an amended version of the Fifth". + +## What the extractor produces today + +Measured on 2026-09-14 and 09-15 with DeepSeek-V3.2, on bases built by today's `dev`. The audit query is in the PR that adds this record. + +| base | entities | facts | facts not bound to the ontology | distinct predicates | +|---|---|---|---|---| +| Blackbaud lease chain (lease bench, 17/18) | 47 | 154 | 70 (45%) | 70 | +| ai-timeline (8 encyclopedia articles, schema.org) | 1,117 | 2,451 | 1,440 (59%) | 1,093 | +| identity bench (12 Chinese notices, forward) | 19 | 25 | 7 | 9 | + +- **Descriptions become entities.** + - ai-timeline: 242 of 1,117 entity names are descriptions ("a datacenter engineer at Google", "academic neuroscientists", "$1.5 billion settlement"). + - The lease has "approximately 12.98 acres of real property located in Berkeley County, South Carolina". Tenant and landlord edges bind to it, with quotes that do not mention it. + - 144 ai-timeline entities hold nothing but their names. +- **Citations leak into names, and aliases are wrong.** + - 11 of 12 lease amendments carry an alias "… dated 2020-xx-xx", and one entity is named "this Seventh Amendment to Lease Agreement". + - ai-timeline records "OpenAI's", "ChatGPT apps", "ChatGPT and other chatbots" and "Microsoft Azure infrastructure" as aliases. + - It also records "deep reinforcement learning" as an alias of "reinforcement learning", which is the pair the governance bench keeps merging wrongly. +- **One thing splits by type label.** AlphaGo is three entities (movie, project, software application); so are Anthropic, GitHub and University College London. +- **Relations are misread.** + - "BLACKBAUD `tenant` 'Lease Agreement'" is written as a literal, backwards. + - Landlord edges point at an amendment. +- **Figures about unnamed things are lost.** The FDA recall bench lost trial sizes and effect percentages in both baseline runs (31/45): the trials have no name, and the span check (#582) drops a subject that is a description. +- **Dates are missing.** 34 of the lease's 154 facts and 167 in ai-timeline have no start although the quote holds a year. Not all of those years are the fact's own. + +The rules the prompt carries have grown to 1, 1a, 1b, 2, 3, 3a, 3c, 4 to 8, 8a to 8h, 9, 10 and 11. Each was written for one failure on one corpus, and they now pull against each other: + +- #705 removed names from facts to save output. The lease bench fell from 13 and 17 to 8 and 8: names were how the server recovered a fact whose handle the model got wrong. +- The span check that keeps "former OpenAI personnel" from becoming OpenAI is what drops every figure about an unnamed trial. +- The server's checks keep growing to catch what one call cannot do: span verdicts, description checks, opening-leak checks, name checks. + +A stronger model does not settle it. With thinking off, on the same code: + +| bench | DeepSeek-V3.2 | deepseek-v4-flash | +|---|---|---| +| FDA approvals recall | 34/45 | 41/45 | +| Chinese statistics and policy recall | 44/50 | 36/50 | +| identity F1 (forward / reverse) | 0.80, 0.77 / 0.69, 0.63 | 0.67 / 0.59 | +| governance agreement, wrong merges | 97.8%, 1 | 97.2%, 2 | + +One run each. The stronger model reads English regulatory text better and is no better at Chinese notices or identity. + +## What mature systems do + +Surveyed on 2026-09-15: GraphRAG, LightRAG, Graphiti, KGGen, iText2KG/ATOM, neo4j-graphrag, LlamaIndex PropertyGraphIndex, LangChain LLMGraphTransformer, Cognee; the canonicalisation literature (EDC, SPIRES/OntoGPT, CESI, CMVC, AutoSchemaKG, ODKE+); entity resolution and linking (GLiNER, ReLiK, Splink, LLM entity matching, ACE/ERE guidelines); temporal extraction (TimeML, HeidelTime, ATOM, Graphiti's timestamps). + +1. **Entities before relations.** An inventory comes first, and a relation's ends must be members of it. KGGen enforces this with a `Literal[...]` over the inventory in structured output ([source](https://github.com/stair-lab/kg-gen/blob/main/src/kg_gen/steps/_2_get_relations.py)); Graphiti drops edges whose ends are not in the node list. +2. **Extraction is loose; convergence happens after.** + - None of these systems holds predicates down with prompt rules. + - EDC extracts open triples, writes a definition for each distinct relation, then canonicalises it against a schema by retrieval and an LLM choice that may answer "none" ([paper](https://arxiv.org/abs/2404.03868)). It reports 0.956 precision against CESI's 0.724. + - SPIRES grounds strings to ontology IDs with a retriever and annotator: 98 of 100 correct, against 3 of 100 when the model writes the IDs itself ([paper](https://academic.oup.com/bioinformatics/article/40/3/btae104/7612230)). +3. **Resolution is layered.** Exact match first, then character or embedding similarity, and an LLM only for what stays ambiguous (Graphiti's MinHash with an entropy gate, [source](https://github.com/getzep/graphiti/blob/main/graphiti_core/utils/maintenance/dedup_helpers.py)). + - Deciding identity from names alone fails structurally. On a cross-script name benchmark, name-only judgements got all 25 namesake cases wrong and 24 right once evidence was given ([paper](https://arxiv.org/html/2608.23507)). +4. **Names and descriptions are separated by structure.** + - ACE and Rich ERE grade mentions as NAM, NOM or PRO. A nominal needs an article, quantifier, possessive or modifier to reach an individual, and an appositive that only states a property is attributive ([ACE guidelines](https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-entities-guidelines-v6.6.pdf)). +5. **Time is normalised outside the extraction call.** + - Graphiti moved date resolution into its own step in v0.29. + - Splitting text along its timeline before extraction took temporal fact F1 from 14.43 to 66.71 ([paper](https://arxiv.org/html/2405.10288)). + - HeidelTime covers 13 languages including Chinese. +6. **Long context does not replace chunking.** Every model in Chroma's context-rot study degrades as input grows ([study](https://www.trychroma.com/research/context-rot)). What chunked systems add is document context: section paths (LightRAG), prior episodes (Graphiti), a per-chunk context line ([Anthropic](https://www.anthropic.com/engineering/contextual-retrieval)). +7. **Temporal invalidation stays local.** Graphiti searches the whole group for contradicted facts. On one production graph, 41% of about 3,950 facts carried `invalid_at`, and three of four sampled were wrong ([issue](https://github.com/getzep/graphiti/issues/1728)). Our engine closes values along one timeline per functional predicate (0022) and stays as it is. + +## Decisions + +### 1. Extraction becomes a pipeline of stages with narrow contracts + +| stage | produces | done by | +|---|---|---| +| 0. Skim | a document card (decision 2) | one LLM call per document, or per major section for long ones | +| 1. Mentions | graded mentions, in-document coreference, the document's entity inventory with names and aliases (decision 3) | LLM per section, with the card | +| 2. Statements | statements in the text's own words, with ends constrained to the inventory, raw time expressions and an evidence sentence (decision 4) | LLM per chunk, with the card and the section path | +| 3. Normalisation | bindings of phrases and type words to the ontology, and proposals for what does not bind (decision 5) | retrieval, then an LLM choice once per distinct phrase per base, then governance | +| 4. Time | validity intervals from time expressions and the card's anchors (decision 6) | deterministic code | +| 5. Identity | entities across documents (decision 7) | layered candidates, scoring, an LLM for the grey zone, batch clustering | +| 6. Ledger | facts, evidence, both clocks, the temporal engine, governance | unchanged | + +- Each stage's output is stored with its evidence, so a later stage can be rerun without rerunning an earlier one. +- A stage that fails leaves the document partly read, with the reason, rather than a document that looks read. + +### 2. The skim produces a document card, not a summary + +The card has fixed fields, and every value carries an evidence span that the server checks against the text: + +- the document's own date and effective date; +- the parties, and the defined terms and who they denote ("Landlord" is HPBB1, LLC; "the Lease" is the agreement dated May 16, 2016); +- the chain of documents it amends or cites, each with its title and date; +- the section tree and its tables; +- the canonical names of the main entities it concerns. + +How the card is used: + +- **It is data.** A document's date, parties and amendment chain go into the ledger from the card. Nothing is prised out of names. +- **It is context.** Stages 1 and 2 receive the card as a shared prefix, so prefix caching applies. It replaces the 1,500-character opening (#681) and the 1,200-character list of known entities. +- **The text prevails.** A chunk that contradicts its card is extracted as the chunk says, and the contradiction is recorded. +- **It is bounded.** It holds these fields and no "key points". A long document gets a document card plus a card per major section. + +### 3. Mentions are graded; descriptions never become entities + +- **Grades.** Stage 1 grades every mention as a name, a description or a pronoun, following ACE/ERE. +- **Descriptions.** A description ("a datacenter engineer at Google", "the second trial", "approximately 12.98 acres of real property") never becomes an entity. What it says becomes a role or attribute of the named entity it belongs to, or, for an unnamed thing, of the named entity it concerns. The trial's size belongs to the drug it tested; the land's acreage belongs to the lease that leases it. +- **Coreference.** It runs within the document. +- **Aliases** come only from name mentions in the same coreference chain, and structure rules the rest out: + - a possessive ("OpenAI's") is not an alias; + - a change of head ("ChatGPT apps", "Microsoft Azure infrastructure") is not an alias; + - a modifier that narrows the reference ("deep reinforcement learning") is not an alias; + - citation text ("dated …", "this …", "as amended") is not part of a name. +- **What the server checks.** It keeps its structural span verdicts (#582) and applies them to entity names and aliases too. It keeps no word lists. + +### 4. Statements use the text's words and name their ends from the inventory + +- **Output contract.** Each statement has: + - its ends, as inventory handles or a literal value; + - the relation in the text's own words, with a one-line definition; + - the raw time expressions it states; + - its evidence sentence. +- **Enforced by structure.** The contract is enforced by a strict tool call whose handle fields are enumerations over the inventory. The schema stays shallow: one statement per array element, so truncation loses a statement rather than a chunk. +- **Reasoning first.** A short reasoning field comes before the answer fields. +- **No ontology in the prompt.** Stage 2 does not see the ontology. A type signature hint may list the few most frequent predicates for the entity types in the chunk; EDC's schema retriever was worth about 0.04 F1. + +### 5. Normalisation happens once per distinct phrase, and what does not bind becomes a proposal + +- **Key.** A binding is keyed by (phrase, definition, type signature) per base and cached. +- **Choice.** + - Candidates come from embedding the definition against the ontology's relations, with each candidate's inverse listed explicitly. + - An LLM answers one of: this key; this key reversed; none. + - Types bind the same way, per entity rather than per mention. +- **History.** A binding is a record with both clocks. A later binding supersedes it, and the facts it covers are rebound; they are not re-extracted. +- **What stays unbound.** + - A phrase that binds to nothing stays on its facts. + - Unbound phrases are clustered, and a cluster becomes an ontology proposal (definition, domain and range, examples, frequency) in the governance queue. + - Proposals are checked against existing properties first, as Wikidata's property process requires. + - A decided proposal writes a binding. +- **Pitfalls to design against:** + - merging adjacent or opposite relations (CESI clustered place of birth with place of death); + - systematic asymmetry in judging inverses; + - lower F1 when the ontology may grow unreviewed. + +### 6. Time expressions are copied by the model and resolved by code + +- **Copying.** Stage 2 copies time expressions verbatim with their spans. +- **Resolution.** Stage 4 resolves them to intervals with precision, anchored on the card's document date and effective date. English uses a rule-based normaliser; Chinese uses one that handles relative expressions and 上年末-style anchors. +- **Three dates.** A document's date, a clause's effective date and an event's date are separate fields. The document date is an anchor for relative expressions, never a default start. + +### 7. Identity is decided on evidence, independent of arrival order + +- **Candidates** are the union of: + - exact normalised names; + - character n-gram similarity; + - romanisation keys; + - cross-script name vectors, mutual nearest across scripts (#709); + - abbreviation keys (the shorter name's characters in order within the longer, numerals equal). + - A type label is a weak feature, never part of the key. +- **Scoring.** + - Fellegi–Sunter-style scoring with term-frequency adjustment lowers the weight of a common name ([Splink](https://moj-analytical-services.github.io/splink/topic_guides/comparisons/term-frequency.html)). + - Its features are context, co-occurring entities, roles and dates. + - Conflicting functional values in overlapping time are cannot-link constraints. So is being declared apart in one response. +- **Grey zone.** It goes to the governor as a choice among candidates, with evidence required ([ComEM](https://arxiv.org/abs/2405.16884)). A name alone never merges. +- **Clustering.** Pair scores are stored as evidenced edges and clustered over the whole base in batch, so the result depends on which documents exist, not on the order they came in. Online resolution attaches provisionally, and the next batch may split it. +- **Measurement.** The identity bench reports the mean and range over shuffled orders. + +### 8. What people and agents review are decisions, and every decision writes back + +The queues change from items to decisions, each shown with its impact: + +| decision | from stage | impact shown | +|---|---|---| +| a phrase's binding, reversal or proposal | 5 | the facts it covers, and the answers that used them | +| an identity cluster and its cannot-links | 7 | the entities and facts it moves | +| a document card's chain, terms or dates | 2 | the documents and facts that rely on it | +| a statement without enough evidence, or one that conflicts | 4, 6 | the fact and its timeline | + +- **Agents first.** The governor decides first (0025, 0043). A person sees what the governor could not decide, what has large impact, and where agents disagree, ordered by impact. +- **Write-back.** A decision writes back as input the pipeline uses deterministically: a binding into the alias table, a split into a cannot-link, a correction into the card. The same question does not return in another shape, and precedents are decisions rather than closed cards. +- **Data model.** `agent_decisions` (target kind, detail, undo) extends to these kinds. +- **Interface.** The interface follows once the prototype shows the volume and shape of real decisions. +- **Calibration.** Calibrating an LLM judge against people happens on labelled bench samples, not in the live queue. + +### 9. Model use follows the stage + +- **Thinking mode** only where reading is hard and the call is rare: the skim. +- **Stages 1 and 2** run with thinking off. +- **Per-stage settings.** A model setting can carry request options such as disabling thinking; deepseek-v4-flash spends about 800 reasoning tokens on a trivial reply. This also serves #690. +- **Local models.** Small local models serve as recall cross-checks, not as the main reader: GLiNER for English mentions; specialised Chinese IE models only as references. Their licences decide whether they can ship at all: ReLiK, REBEL, Maverick and IEPile are non-commercial, HanLP's Chinese models are research-only, LTP is paid for commercial use, and Zingg is AGPL. + +### 10. Today's extractor is frozen while the prototype is measured + +- No new prompt rules land on the current extractor. +- Structural fixes already merged stay: the per-chunk budget (#704) and governance changes. +- Open cuts that add prompt rules wait: an unnamed trial's figures, grant receivers, the romanisation rule in #709. + +## Not doing + +- **Untyped relations with descriptions only** (GraphRAG, LightRAG). They avoid predicate explosion by giving up the ontology, which is what this product governs. +- **Graphiti's group-wide contradiction search.** The temporal engine stays local to a timeline. +- **Grammar-constrained decoding against the whole ontology.** It forces wrong choices, favours empty output and is slow ([GenIE analysis](https://arxiv.org/html/2305.13971v6)). +- **One-shot whole-document extraction on long-context models.** Quality decays with length, and the card gives stage 2 the context without it. +- **A lexicon of honorifics, suffixes or citation words.** Decisions 3 and 7 use structure; semantics stay in prompts. + +## Migration + +1. **An offline prototype** runs stages 0 and 3 as scripts against today's extraction output on five benches: + - lease chain; + - FDA approvals; + - Chinese statistics and policy; + - NVDA filings; + - ai-timeline. + + It uses the same model endpoint, with no server changes. +2. **Stages 1 and 2** join the prototype. Stage 0's card feeds them. +3. **In the server, behind a base setting** (old path by default), in order: card and time (0, 4), normalisation (3), mentions and statements (1, 2), batch identity (7). +4. **The decision queues and their interface** come last. + +Each step lands only if every bench holds within its variance, and the audit metrics below improve. + +## Measurement + +- **Recall benches.** Every stage is measured on the five recall benches (`recall.mjs --corpus`, `lease_bench.py`), the identity bench over shuffled orders, and the governance bench. +- **Audit metrics** on the resulting bases: + - distinct predicates per fact, and the unbound share; + - description entities and names-only entities; + - aliases that fail the structural checks; + - entities split across type labels; + - facts without a start whose evidence states a date; + - statements whose ends do not appear in their evidence. +- **Normalisation bench.** Gold (phrase, context) → ontology key, reversed, new, or none. It measures binding precision, binding rate and direction accuracy, reported by frequency band. +- **Precision.** Precision needs sampled human labels. An LLM judge is used only after its agreement with those labels is known. +- **Cost.** Tokens and latency per document per stage, against today's single call. + +## Open questions + +- **Chinese coreference and time.** The mature open tools are English-first. Whether stage 1 coreference and stage 4 normalisation for Chinese are LLM calls or rule libraries is measured on the Chinese benches. +- **Atomic facts.** ATOM's decontextualised atomic facts raised fact recall by 31% and cost 9% precision ([paper](https://arxiv.org/html/2510.22590v2)). Whether stage 2 needs that layer is measured, not assumed. +- **XBRL.** Figures in SEC filings exist as structured facts; whether filings bypass stage 2 for them is a separate record. +- **Card cost on long documents.** It is measured on the 136-chunk earnings release before a section-card threshold is chosen. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 8dca1b8ea..933e4f505 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -66,6 +66,7 @@ The test for writing one: if someone (including us) looks at a piece of code in | 0040 | [A chunk says where its words came from](0040-a-chunk-says-where-its-words-came-from.md) | Cuts 1–3 implemented (the ledger shape, and a file that needs a reader degrades with an alert instead of becoming garbage text; scans and images read by a workspace's MinerU service, one segment per page, waiting on the service without spending retries; recordings read by a diarizing transcription model, speakers in the text and times in the anchor, an unlabelled transcript degrades) · scans and images through a MinerU service, recordings only with speaker labels (revised 2026-09-15) · a chunk carries an **origin** (`stated`, `ocr`, `transcribed`, `described`) and the model that produced it, and an **anchor** back into the original bytes (a page and region, a time range, an image inside the file) — decided before any media reader, because a transcript stored without its times can never be tied to the recording again · the packer never mixes origins in a chunk · facts from a description enter below `AUTO_CLOSE_MIN_CONFIDENCE`, so a misread chart opens a conflict instead of closing a correct fact · per-modality model settings, media reading as a resumable job, origin and anchor in the API, MCP and RDF · cuts: ledger shape, scans via MinerU, recordings, descriptions, video | | 0041 | [A name is a claim about an entity](0041-a-name-is-a-claim-about-an-entity.md) | Cut 0 built (identity bench) · cut 1 implemented (#670): names are value facts on `known_as`, the extractor reports other names, a shared name goes to the adjudicator; forward/reverse F1 0.43/0.54 → 0.68/0.68 · cuts 2–4 (name vectors and neighbours, evidence decides, re-evaluation) not started | | 0042 | [The chat loop is a runner with hooks](0042-the-chat-loop-is-a-runner-with-hooks.md) | Implemented (#548) · the loop is rig's runner and every policy is a hook with a typed result · the wire stays `LlmClient` behind `RigModel` · a turn cannot end before a tool has run, `no_evidence_needed` is the exit for questions not about the base · RAG fallback only on a 400/422 to the first request with tools · an empty reply is asked again once · the skip rate is the model's (DeepSeek-V3 1–3 of 12, Qwen2.5-72B 0) and recorded, not prevented | +| 0044 | [A document is read in stages](0044-a-document-is-read-in-stages.md) | Proposed · nothing built · extraction becomes stages with narrow contracts: a skim that writes a document card with evidenced fields, graded mentions where descriptions never become entities, statements in the text's words with ends from the inventory, normalisation once per distinct phrase with proposals through governance, time resolved by code, identity by evidence in batch · reviews become decisions that write back · today's extractor frozen until an offline prototype of stages 0 and 3 reports on five benches | ## Not a decision record diff --git a/scripts/bench/audit.sql b/scripts/bench/audit.sql new file mode 100644 index 000000000..ab13cd7dd --- /dev/null +++ b/scripts/bench/audit.sql @@ -0,0 +1,90 @@ +-- 抽取质量审计（诊断用，启发式）：psql -v kb= +\set QUIET on +\pset footer off +\pset border 0 +CREATE TEMP VIEW ent AS + SELECT e.id, e.canonical_name AS name, t.key AS type + FROM entities e LEFT JOIN entity_types t ON t.id = e.type_id + WHERE e.kb_id = :'kb' AND e.merged_into IS NULL; +CREATE TEMP VIEW fx AS + SELECT f.id, f.subject_id, f.object_id, s.canonical_name AS subj, o.canonical_name AS obj, + f.object_value #>> '{value}' AS val, f.object_value #>> '{}' AS raw_val, + coalesce(rt.key, (SELECT fe.proposed_predicate FROM fact_evidence fe WHERE fe.fact_id = f.id LIMIT 1)) AS pred, + rt.id IS NOT NULL AS bound, rt.key = 'known_as' AS is_name, + f.valid_from, f.valid_to, f.confidence, + (SELECT fe.quote FROM fact_evidence fe WHERE fe.fact_id = f.id AND fe.quote IS NOT NULL LIMIT 1) AS quote + FROM facts f + JOIN entities s ON s.id = f.subject_id + LEFT JOIN entities o ON o.id = f.object_id + LEFT JOIN relation_types rt ON rt.id = f.predicate_id + WHERE f.kb_id = :'kb' AND f.invalidated_at IS NULL; + +\echo '== 总量' +SELECT (SELECT count(*) FROM ent) AS entities, + (SELECT count(*) FROM fx WHERE NOT coalesce(is_name, false)) AS facts, + (SELECT count(*) FROM fx WHERE NOT coalesce(is_name, false) AND NOT bound) AS unbound_facts, + (SELECT count(DISTINCT pred) FROM fx WHERE NOT coalesce(is_name, false)) AS distinct_predicates; + +\echo '== A 名字像描述：小写开头 / 含数字+小写词 / 超过 7 个词 / 带 this·that·such·dated·as amended' +SELECT name, type FROM ent + WHERE name ~ '^[a-z]' OR name ~ '[0-9].* [a-z]{3,}' OR array_length(regexp_split_to_array(name, '\s+'), 1) > 7 + OR name ~* '^(this|that|such|said|the said)\s' OR name ~* '\s(dated|as amended)\b' + ORDER BY name LIMIT 25; +SELECT count(*) AS a_count FROM ent + WHERE name ~ '^[a-z]' OR name ~ '[0-9].* [a-z]{3,}' OR array_length(regexp_split_to_array(name, '\s+'), 1) > 7 + OR name ~* '^(this|that|such|said|the said)\s' OR name ~* '\s(dated|as amended)\b'; + +\echo '== B 别名比本名长且包含本名（可能是下属/描述，不是别名）' +SELECT e.name, f.val AS alias FROM fx f JOIN ent e ON e.id = f.subject_id + WHERE f.is_name AND length(f.val) > length(e.name) AND position(lower(e.name) IN lower(f.val)) > 0 + LIMIT 15; + +\echo '== C 大小写/空白归一后同名的多个实体' +SELECT lower(regexp_replace(name, '\s+', ' ', 'g')) AS norm, count(*), string_agg(DISTINCT coalesce(type,'-'), ',') AS types + FROM ent GROUP BY 1 HAVING count(*) > 1 ORDER BY 2 DESC LIMIT 15; + +\echo '== D 一个名字是另一个名字 + 尾巴（dated/of/the/逗号…）且同类型' +SELECT a.name AS shorter, b.name AS longer FROM ent a JOIN ent b ON a.id <> b.id AND coalesce(a.type,'') = coalesce(b.type,'') + WHERE length(b.name) > length(a.name) + 3 AND lower(b.name) LIKE lower(a.name) || ' %' + LIMIT 20; + +\echo '== E 值事实：值（去掉空白与逗号）不在引文里' +SELECT subj, pred, left(val, 40) AS val, left(quote, 90) AS quote FROM fx + WHERE NOT coalesce(is_name, false) AND val IS NOT NULL AND quote IS NOT NULL + AND position(lower(regexp_replace(val, '[\s,]', '', 'g')) IN lower(regexp_replace(quote, '[\s,]', '', 'g'))) = 0 + AND val !~ '^\d{4}-\d{2}(-\d{2})?$' + LIMIT 15; +SELECT count(*) AS e_count FROM fx + WHERE NOT coalesce(is_name, false) AND val IS NOT NULL AND quote IS NOT NULL + AND position(lower(regexp_replace(val, '[\s,]', '', 'g')) IN lower(regexp_replace(quote, '[\s,]', '', 'g'))) = 0 + AND val !~ '^\d{4}-\d{2}(-\d{2})?$'; + +\echo '== F 关系事实：宾语名字的首词不在引文里（可能挂错了宾语）' +SELECT subj, pred, obj, left(quote, 90) AS quote FROM fx + WHERE object_id IS NOT NULL AND quote IS NOT NULL + AND position(lower(split_part(obj, ' ', 1)) IN lower(quote)) = 0 + LIMIT 15; +SELECT count(*) AS f_count, (SELECT count(*) FROM fx WHERE object_id IS NOT NULL) AS relation_facts FROM fx + WHERE object_id IS NOT NULL AND quote IS NOT NULL + AND position(lower(split_part(obj, ' ', 1)) IN lower(quote)) = 0; + +\echo '== G 关系事实：主语名字的首词也不在引文里' +SELECT count(*) AS g_count FROM fx + WHERE NOT coalesce(is_name, false) AND quote IS NOT NULL + AND position(lower(split_part(subj, ' ', 1)) IN lower(quote)) = 0; + +\echo '== H 没有起点、但引文里写着年份的状态事实' +SELECT subj, pred, coalesce(obj, left(val, 30)) AS what, left(quote, 90) AS quote FROM fx + WHERE NOT coalesce(is_name, false) AND valid_from IS NULL AND quote ~ '(19|20)\d{2}' + LIMIT 10; +SELECT count(*) AS h_count FROM fx WHERE NOT coalesce(is_name, false) AND valid_from IS NULL AND quote ~ '(19|20)\d{2}'; + +\echo '== I 最常见的未绑定谓词（说法泛滥/同义词）' +SELECT pred, count(*) FROM fx WHERE NOT coalesce(is_name, false) AND NOT bound GROUP BY 1 ORDER BY 2 DESC LIMIT 15; + +\echo '== J 孤立实体（除名字外没有任何事实）' +SELECT count(*) AS orphan_entities FROM ent e + WHERE NOT EXISTS (SELECT 1 FROM fx f WHERE (f.subject_id = e.id OR f.object_id = e.id) AND NOT coalesce(f.is_name, false)); +SELECT name, type FROM ent e + WHERE NOT EXISTS (SELECT 1 FROM fx f WHERE (f.subject_id = e.id OR f.object_id = e.id) AND NOT coalesce(f.is_name, false)) + LIMIT 12; From ede6d46232dca135ae34fcd8299df94e09093c0a Mon Sep 17 00:00:00 2001 From: WaylandYang Date: Tue, 15 Sep 2026 14:19:16 +0800 Subject: [PATCH 2/7] 0044 says which builds the model comparison ran on Co-Authored-By: Claude Opus 5 Signed-off-by: WaylandYang --- docs/decisions/0044-a-document-is-read-in-stages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/0044-a-document-is-read-in-stages.md b/docs/decisions/0044-a-document-is-read-in-stages.md index de9af58a4..8f46517b4 100644 --- a/docs/decisions/0044-a-document-is-read-in-stages.md +++ b/docs/decisions/0044-a-document-is-read-in-stages.md @@ -54,7 +54,7 @@ The rules the prompt carries have grown to 1, 1a, 1b, 2, 3, 3a, 3c, 4 to 8, 8a t - The span check that keeps "former OpenAI personnel" from becoming OpenAI is what drops every figure about an unnamed trial. - The server's checks keep growing to catch what one call cannot do: span verdicts, description checks, opening-leak checks, name checks. -A stronger model does not settle it. With thinking off, on the same code: +A stronger model does not settle it. With thinking off, on the same extraction code (the identity and governance rows ran on builds that differ only in how name vectors pair, #709): | bench | DeepSeek-V3.2 | deepseek-v4-flash | |---|---|---| From c8bffdd782322b362e468cca1b986521173658ec Mon Sep 17 00:00:00 2001 From: WaylandYang Date: Tue, 15 Sep 2026 14:28:22 +0800 Subject: [PATCH 3/7] 0044 keeps lexicons as governed data Co-Authored-By: Claude Opus 5 Signed-off-by: WaylandYang --- docs/decisions/0044-a-document-is-read-in-stages.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/decisions/0044-a-document-is-read-in-stages.md b/docs/decisions/0044-a-document-is-read-in-stages.md index 8f46517b4..59732179b 100644 --- a/docs/decisions/0044-a-document-is-read-in-stages.md +++ b/docs/decisions/0044-a-document-is-read-in-stages.md @@ -150,6 +150,7 @@ How the card is used: - An LLM answers one of: this key; this key reversed; none. - Types bind the same way, per entity rather than per mention. - **History.** A binding is a record with both clocks. A later binding supersedes it, and the facts it covers are rebound; they are not re-extracted. +- **The binding table is a governed lexicon.** "is_part_of", "隶属于" and "subsidiary of" → `part_of`, with direction. It is data in the base, with a source and both clocks, and a person can revert an entry. It is not a list in code. - **What stays unbound.** - A phrase that binds to nothing stays on its facts. - Unbound phrases are clustered, and a cluster becomes an ontology proposal (definition, domain and range, examples, frequency) in the governance queue. @@ -181,6 +182,11 @@ How the card is used: - Conflicting functional values in overlapping time are cannot-link constraints. So is being declared apart in one response. - **Grey zone.** It goes to the governor as a choice among candidates, with evidence required ([ComEM](https://arxiv.org/abs/2405.16884)). A name alone never merges. - **Clustering.** Pair scores are stored as evidenced edges and clustered over the whole base in batch, so the result depends on which documents exist, not on the order they came in. Online resolution attaches provisionally, and the next batch may split it. +- **Lexicons are governed data.** + - **Alias table:** 海探1 = 海洋探测器1号, 英伟达 = NVIDIA, from what the text states and from merge decisions. + - **Cannot-link list:** from declarations and from decisions to keep apart. + - **External gazetteers (optional):** drug names, listed companies, administrative divisions, Wikidata aliases. Each is imported with its source and version and switched on per base. + - All of these live in the base with evidence and can be reverted. - **Measurement.** The identity bench reports the mean and range over shuffled orders. ### 8. What people and agents review are decisions, and every decision writes back @@ -215,11 +221,11 @@ The queues change from items to decisions, each shown with its impact: ## Not doing -- **Untyped relations with descriptions only** (GraphRAG, LightRAG). They avoid predicate explosion by giving up the ontology, which is what this product governs. -- **Graphiti's group-wide contradiction search.** The temporal engine stays local to a timeline. +- **Untyped relations with descriptions only** (GraphRAG, LightRAG). A relation is stored as a sentence and a strength, with no predicate. They avoid predicate explosion by giving up the ontology, and with it the axioms: functionality and inverses. Those are what the temporal engine and reasoning run on. +- **Graphiti's group-wide contradiction search.** A new fact searches the whole graph for semantically related facts, and an LLM marks the contradicted ones invalid. Related but independent facts get invalidated. The temporal engine stays local: a value closes only on its own subject's timeline for a functional predicate. - **Grammar-constrained decoding against the whole ontology.** It forces wrong choices, favours empty output and is slow ([GenIE analysis](https://arxiv.org/html/2305.13971v6)). - **One-shot whole-document extraction on long-context models.** Quality decays with length, and the card gives stage 2 the context without it. -- **A lexicon of honorifics, suffixes or citation words.** Decisions 3 and 7 use structure; semantics stay in prompts. +- **Word lists written in code** of honorifics, company suffixes or citation words that rules match against (#637's suffix list was withdrawn for this). Grammatical judgements such as possessives and determiners belong to the mention grading in stage 1. Lexicons that are data — bindings, aliases, cannot-links, gazetteers — are part of the design (decisions 5 and 7). ## Migration From 430aef389e4832a7472b2c21ff6a8cb3fd5431ca Mon Sep 17 00:00:00 2001 From: WaylandYang Date: Wed, 16 Sep 2026 01:20:04 +0800 Subject: [PATCH 4/7] 0044 makes the ontology a view over what documents say Co-Authored-By: Claude Opus 5 Signed-off-by: WaylandYang --- .../0044-a-document-is-read-in-stages.md | 265 ------------------ ...ology-is-a-view-over-what-documents-say.md | 123 ++++++++ docs/decisions/README.md | 1 + 3 files changed, 124 insertions(+), 265 deletions(-) delete mode 100644 docs/decisions/0044-a-document-is-read-in-stages.md create mode 100644 docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md diff --git a/docs/decisions/0044-a-document-is-read-in-stages.md b/docs/decisions/0044-a-document-is-read-in-stages.md deleted file mode 100644 index 59732179b..000000000 --- a/docs/decisions/0044-a-document-is-read-in-stages.md +++ /dev/null @@ -1,265 +0,0 @@ -# 0044 · A document is read in stages - -- **Status**: Proposed · nothing built · the next step is an offline prototype of stages 0 and 3, measured on five benches against today's extractor · prompt rules for today's extractor are frozen until it reports -- **Written**: 2026-09-15 (conventions in the [README](README.md)) -- **Related**: - - Evidence that motivated this record: - - [0006](0006-ontology-scale-and-the-prompt.md): the ontology slice in the prompt. - - [0039](0039-a-chunk-is-what-extraction-sees.md): chunks as the unit of extraction. - - [0041](0041-a-name-is-a-claim-about-an-entity.md): names as facts and the identity bench. Its cuts 2–4 fold into decision 7. - - [0022](0022-an-unknown-date-is-not-an-open-one.md): the temporal engine this record keeps. - - #701: the per-chunk ontology budget. - - #705, closed: sentence citations; names carry recovery. - - #709: cross-script name vectors. - - Records this one extends: - - [0025](0025-governance-reads-the-ledger-before-it-decides.md) and [0043](0043-every-review-queue-is-governed.md): governance of every queue. Decision 8 changes what the queues hold. - - [0003](0003-ontology-growth-loop.md): the ontology grows out of the corpus. Decision 5 is where it grows from. - -> A lease signed in 2016 is amended eleven times. Each amendment cites the lease and its earlier amendments by title and date. Today every 1,000-character chunk of every amendment asks one model call for all of the following at once: -> -> - the entities, their names, aliases and types, chosen from 200 candidate classes; -> - the facts, their predicates chosen from 180 candidate relations, and their direction; -> - values, units, validity dates, quotes and the words that name each side. -> -> The chunk cannot see the rest of the document. "The Fifth Amendment dated February 18, 2020" comes back as its own entity, with its date inside its name. A cited list of amendments comes back as "the Ninth Amendment is an amended version of the Fifth". - -## What the extractor produces today - -Measured on 2026-09-14 and 09-15 with DeepSeek-V3.2, on bases built by today's `dev`. The audit query is in the PR that adds this record. - -| base | entities | facts | facts not bound to the ontology | distinct predicates | -|---|---|---|---|---| -| Blackbaud lease chain (lease bench, 17/18) | 47 | 154 | 70 (45%) | 70 | -| ai-timeline (8 encyclopedia articles, schema.org) | 1,117 | 2,451 | 1,440 (59%) | 1,093 | -| identity bench (12 Chinese notices, forward) | 19 | 25 | 7 | 9 | - -- **Descriptions become entities.** - - ai-timeline: 242 of 1,117 entity names are descriptions ("a datacenter engineer at Google", "academic neuroscientists", "$1.5 billion settlement"). - - The lease has "approximately 12.98 acres of real property located in Berkeley County, South Carolina". Tenant and landlord edges bind to it, with quotes that do not mention it. - - 144 ai-timeline entities hold nothing but their names. -- **Citations leak into names, and aliases are wrong.** - - 11 of 12 lease amendments carry an alias "… dated 2020-xx-xx", and one entity is named "this Seventh Amendment to Lease Agreement". - - ai-timeline records "OpenAI's", "ChatGPT apps", "ChatGPT and other chatbots" and "Microsoft Azure infrastructure" as aliases. - - It also records "deep reinforcement learning" as an alias of "reinforcement learning", which is the pair the governance bench keeps merging wrongly. -- **One thing splits by type label.** AlphaGo is three entities (movie, project, software application); so are Anthropic, GitHub and University College London. -- **Relations are misread.** - - "BLACKBAUD `tenant` 'Lease Agreement'" is written as a literal, backwards. - - Landlord edges point at an amendment. -- **Figures about unnamed things are lost.** The FDA recall bench lost trial sizes and effect percentages in both baseline runs (31/45): the trials have no name, and the span check (#582) drops a subject that is a description. -- **Dates are missing.** 34 of the lease's 154 facts and 167 in ai-timeline have no start although the quote holds a year. Not all of those years are the fact's own. - -The rules the prompt carries have grown to 1, 1a, 1b, 2, 3, 3a, 3c, 4 to 8, 8a to 8h, 9, 10 and 11. Each was written for one failure on one corpus, and they now pull against each other: - -- #705 removed names from facts to save output. The lease bench fell from 13 and 17 to 8 and 8: names were how the server recovered a fact whose handle the model got wrong. -- The span check that keeps "former OpenAI personnel" from becoming OpenAI is what drops every figure about an unnamed trial. -- The server's checks keep growing to catch what one call cannot do: span verdicts, description checks, opening-leak checks, name checks. - -A stronger model does not settle it. With thinking off, on the same extraction code (the identity and governance rows ran on builds that differ only in how name vectors pair, #709): - -| bench | DeepSeek-V3.2 | deepseek-v4-flash | -|---|---|---| -| FDA approvals recall | 34/45 | 41/45 | -| Chinese statistics and policy recall | 44/50 | 36/50 | -| identity F1 (forward / reverse) | 0.80, 0.77 / 0.69, 0.63 | 0.67 / 0.59 | -| governance agreement, wrong merges | 97.8%, 1 | 97.2%, 2 | - -One run each. The stronger model reads English regulatory text better and is no better at Chinese notices or identity. - -## What mature systems do - -Surveyed on 2026-09-15: GraphRAG, LightRAG, Graphiti, KGGen, iText2KG/ATOM, neo4j-graphrag, LlamaIndex PropertyGraphIndex, LangChain LLMGraphTransformer, Cognee; the canonicalisation literature (EDC, SPIRES/OntoGPT, CESI, CMVC, AutoSchemaKG, ODKE+); entity resolution and linking (GLiNER, ReLiK, Splink, LLM entity matching, ACE/ERE guidelines); temporal extraction (TimeML, HeidelTime, ATOM, Graphiti's timestamps). - -1. **Entities before relations.** An inventory comes first, and a relation's ends must be members of it. KGGen enforces this with a `Literal[...]` over the inventory in structured output ([source](https://github.com/stair-lab/kg-gen/blob/main/src/kg_gen/steps/_2_get_relations.py)); Graphiti drops edges whose ends are not in the node list. -2. **Extraction is loose; convergence happens after.** - - None of these systems holds predicates down with prompt rules. - - EDC extracts open triples, writes a definition for each distinct relation, then canonicalises it against a schema by retrieval and an LLM choice that may answer "none" ([paper](https://arxiv.org/abs/2404.03868)). It reports 0.956 precision against CESI's 0.724. - - SPIRES grounds strings to ontology IDs with a retriever and annotator: 98 of 100 correct, against 3 of 100 when the model writes the IDs itself ([paper](https://academic.oup.com/bioinformatics/article/40/3/btae104/7612230)). -3. **Resolution is layered.** Exact match first, then character or embedding similarity, and an LLM only for what stays ambiguous (Graphiti's MinHash with an entropy gate, [source](https://github.com/getzep/graphiti/blob/main/graphiti_core/utils/maintenance/dedup_helpers.py)). - - Deciding identity from names alone fails structurally. On a cross-script name benchmark, name-only judgements got all 25 namesake cases wrong and 24 right once evidence was given ([paper](https://arxiv.org/html/2608.23507)). -4. **Names and descriptions are separated by structure.** - - ACE and Rich ERE grade mentions as NAM, NOM or PRO. A nominal needs an article, quantifier, possessive or modifier to reach an individual, and an appositive that only states a property is attributive ([ACE guidelines](https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-entities-guidelines-v6.6.pdf)). -5. **Time is normalised outside the extraction call.** - - Graphiti moved date resolution into its own step in v0.29. - - Splitting text along its timeline before extraction took temporal fact F1 from 14.43 to 66.71 ([paper](https://arxiv.org/html/2405.10288)). - - HeidelTime covers 13 languages including Chinese. -6. **Long context does not replace chunking.** Every model in Chroma's context-rot study degrades as input grows ([study](https://www.trychroma.com/research/context-rot)). What chunked systems add is document context: section paths (LightRAG), prior episodes (Graphiti), a per-chunk context line ([Anthropic](https://www.anthropic.com/engineering/contextual-retrieval)). -7. **Temporal invalidation stays local.** Graphiti searches the whole group for contradicted facts. On one production graph, 41% of about 3,950 facts carried `invalid_at`, and three of four sampled were wrong ([issue](https://github.com/getzep/graphiti/issues/1728)). Our engine closes values along one timeline per functional predicate (0022) and stays as it is. - -## Decisions - -### 1. Extraction becomes a pipeline of stages with narrow contracts - -| stage | produces | done by | -|---|---|---| -| 0. Skim | a document card (decision 2) | one LLM call per document, or per major section for long ones | -| 1. Mentions | graded mentions, in-document coreference, the document's entity inventory with names and aliases (decision 3) | LLM per section, with the card | -| 2. Statements | statements in the text's own words, with ends constrained to the inventory, raw time expressions and an evidence sentence (decision 4) | LLM per chunk, with the card and the section path | -| 3. Normalisation | bindings of phrases and type words to the ontology, and proposals for what does not bind (decision 5) | retrieval, then an LLM choice once per distinct phrase per base, then governance | -| 4. Time | validity intervals from time expressions and the card's anchors (decision 6) | deterministic code | -| 5. Identity | entities across documents (decision 7) | layered candidates, scoring, an LLM for the grey zone, batch clustering | -| 6. Ledger | facts, evidence, both clocks, the temporal engine, governance | unchanged | - -- Each stage's output is stored with its evidence, so a later stage can be rerun without rerunning an earlier one. -- A stage that fails leaves the document partly read, with the reason, rather than a document that looks read. - -### 2. The skim produces a document card, not a summary - -The card has fixed fields, and every value carries an evidence span that the server checks against the text: - -- the document's own date and effective date; -- the parties, and the defined terms and who they denote ("Landlord" is HPBB1, LLC; "the Lease" is the agreement dated May 16, 2016); -- the chain of documents it amends or cites, each with its title and date; -- the section tree and its tables; -- the canonical names of the main entities it concerns. - -How the card is used: - -- **It is data.** A document's date, parties and amendment chain go into the ledger from the card. Nothing is prised out of names. -- **It is context.** Stages 1 and 2 receive the card as a shared prefix, so prefix caching applies. It replaces the 1,500-character opening (#681) and the 1,200-character list of known entities. -- **The text prevails.** A chunk that contradicts its card is extracted as the chunk says, and the contradiction is recorded. -- **It is bounded.** It holds these fields and no "key points". A long document gets a document card plus a card per major section. - -### 3. Mentions are graded; descriptions never become entities - -- **Grades.** Stage 1 grades every mention as a name, a description or a pronoun, following ACE/ERE. -- **Descriptions.** A description ("a datacenter engineer at Google", "the second trial", "approximately 12.98 acres of real property") never becomes an entity. What it says becomes a role or attribute of the named entity it belongs to, or, for an unnamed thing, of the named entity it concerns. The trial's size belongs to the drug it tested; the land's acreage belongs to the lease that leases it. -- **Coreference.** It runs within the document. -- **Aliases** come only from name mentions in the same coreference chain, and structure rules the rest out: - - a possessive ("OpenAI's") is not an alias; - - a change of head ("ChatGPT apps", "Microsoft Azure infrastructure") is not an alias; - - a modifier that narrows the reference ("deep reinforcement learning") is not an alias; - - citation text ("dated …", "this …", "as amended") is not part of a name. -- **What the server checks.** It keeps its structural span verdicts (#582) and applies them to entity names and aliases too. It keeps no word lists. - -### 4. Statements use the text's words and name their ends from the inventory - -- **Output contract.** Each statement has: - - its ends, as inventory handles or a literal value; - - the relation in the text's own words, with a one-line definition; - - the raw time expressions it states; - - its evidence sentence. -- **Enforced by structure.** The contract is enforced by a strict tool call whose handle fields are enumerations over the inventory. The schema stays shallow: one statement per array element, so truncation loses a statement rather than a chunk. -- **Reasoning first.** A short reasoning field comes before the answer fields. -- **No ontology in the prompt.** Stage 2 does not see the ontology. A type signature hint may list the few most frequent predicates for the entity types in the chunk; EDC's schema retriever was worth about 0.04 F1. - -### 5. Normalisation happens once per distinct phrase, and what does not bind becomes a proposal - -- **Key.** A binding is keyed by (phrase, definition, type signature) per base and cached. -- **Choice.** - - Candidates come from embedding the definition against the ontology's relations, with each candidate's inverse listed explicitly. - - An LLM answers one of: this key; this key reversed; none. - - Types bind the same way, per entity rather than per mention. -- **History.** A binding is a record with both clocks. A later binding supersedes it, and the facts it covers are rebound; they are not re-extracted. -- **The binding table is a governed lexicon.** "is_part_of", "隶属于" and "subsidiary of" → `part_of`, with direction. It is data in the base, with a source and both clocks, and a person can revert an entry. It is not a list in code. -- **What stays unbound.** - - A phrase that binds to nothing stays on its facts. - - Unbound phrases are clustered, and a cluster becomes an ontology proposal (definition, domain and range, examples, frequency) in the governance queue. - - Proposals are checked against existing properties first, as Wikidata's property process requires. - - A decided proposal writes a binding. -- **Pitfalls to design against:** - - merging adjacent or opposite relations (CESI clustered place of birth with place of death); - - systematic asymmetry in judging inverses; - - lower F1 when the ontology may grow unreviewed. - -### 6. Time expressions are copied by the model and resolved by code - -- **Copying.** Stage 2 copies time expressions verbatim with their spans. -- **Resolution.** Stage 4 resolves them to intervals with precision, anchored on the card's document date and effective date. English uses a rule-based normaliser; Chinese uses one that handles relative expressions and 上年末-style anchors. -- **Three dates.** A document's date, a clause's effective date and an event's date are separate fields. The document date is an anchor for relative expressions, never a default start. - -### 7. Identity is decided on evidence, independent of arrival order - -- **Candidates** are the union of: - - exact normalised names; - - character n-gram similarity; - - romanisation keys; - - cross-script name vectors, mutual nearest across scripts (#709); - - abbreviation keys (the shorter name's characters in order within the longer, numerals equal). - - A type label is a weak feature, never part of the key. -- **Scoring.** - - Fellegi–Sunter-style scoring with term-frequency adjustment lowers the weight of a common name ([Splink](https://moj-analytical-services.github.io/splink/topic_guides/comparisons/term-frequency.html)). - - Its features are context, co-occurring entities, roles and dates. - - Conflicting functional values in overlapping time are cannot-link constraints. So is being declared apart in one response. -- **Grey zone.** It goes to the governor as a choice among candidates, with evidence required ([ComEM](https://arxiv.org/abs/2405.16884)). A name alone never merges. -- **Clustering.** Pair scores are stored as evidenced edges and clustered over the whole base in batch, so the result depends on which documents exist, not on the order they came in. Online resolution attaches provisionally, and the next batch may split it. -- **Lexicons are governed data.** - - **Alias table:** 海探1 = 海洋探测器1号, 英伟达 = NVIDIA, from what the text states and from merge decisions. - - **Cannot-link list:** from declarations and from decisions to keep apart. - - **External gazetteers (optional):** drug names, listed companies, administrative divisions, Wikidata aliases. Each is imported with its source and version and switched on per base. - - All of these live in the base with evidence and can be reverted. -- **Measurement.** The identity bench reports the mean and range over shuffled orders. - -### 8. What people and agents review are decisions, and every decision writes back - -The queues change from items to decisions, each shown with its impact: - -| decision | from stage | impact shown | -|---|---|---| -| a phrase's binding, reversal or proposal | 5 | the facts it covers, and the answers that used them | -| an identity cluster and its cannot-links | 7 | the entities and facts it moves | -| a document card's chain, terms or dates | 2 | the documents and facts that rely on it | -| a statement without enough evidence, or one that conflicts | 4, 6 | the fact and its timeline | - -- **Agents first.** The governor decides first (0025, 0043). A person sees what the governor could not decide, what has large impact, and where agents disagree, ordered by impact. -- **Write-back.** A decision writes back as input the pipeline uses deterministically: a binding into the alias table, a split into a cannot-link, a correction into the card. The same question does not return in another shape, and precedents are decisions rather than closed cards. -- **Data model.** `agent_decisions` (target kind, detail, undo) extends to these kinds. -- **Interface.** The interface follows once the prototype shows the volume and shape of real decisions. -- **Calibration.** Calibrating an LLM judge against people happens on labelled bench samples, not in the live queue. - -### 9. Model use follows the stage - -- **Thinking mode** only where reading is hard and the call is rare: the skim. -- **Stages 1 and 2** run with thinking off. -- **Per-stage settings.** A model setting can carry request options such as disabling thinking; deepseek-v4-flash spends about 800 reasoning tokens on a trivial reply. This also serves #690. -- **Local models.** Small local models serve as recall cross-checks, not as the main reader: GLiNER for English mentions; specialised Chinese IE models only as references. Their licences decide whether they can ship at all: ReLiK, REBEL, Maverick and IEPile are non-commercial, HanLP's Chinese models are research-only, LTP is paid for commercial use, and Zingg is AGPL. - -### 10. Today's extractor is frozen while the prototype is measured - -- No new prompt rules land on the current extractor. -- Structural fixes already merged stay: the per-chunk budget (#704) and governance changes. -- Open cuts that add prompt rules wait: an unnamed trial's figures, grant receivers, the romanisation rule in #709. - -## Not doing - -- **Untyped relations with descriptions only** (GraphRAG, LightRAG). A relation is stored as a sentence and a strength, with no predicate. They avoid predicate explosion by giving up the ontology, and with it the axioms: functionality and inverses. Those are what the temporal engine and reasoning run on. -- **Graphiti's group-wide contradiction search.** A new fact searches the whole graph for semantically related facts, and an LLM marks the contradicted ones invalid. Related but independent facts get invalidated. The temporal engine stays local: a value closes only on its own subject's timeline for a functional predicate. -- **Grammar-constrained decoding against the whole ontology.** It forces wrong choices, favours empty output and is slow ([GenIE analysis](https://arxiv.org/html/2305.13971v6)). -- **One-shot whole-document extraction on long-context models.** Quality decays with length, and the card gives stage 2 the context without it. -- **Word lists written in code** of honorifics, company suffixes or citation words that rules match against (#637's suffix list was withdrawn for this). Grammatical judgements such as possessives and determiners belong to the mention grading in stage 1. Lexicons that are data — bindings, aliases, cannot-links, gazetteers — are part of the design (decisions 5 and 7). - -## Migration - -1. **An offline prototype** runs stages 0 and 3 as scripts against today's extraction output on five benches: - - lease chain; - - FDA approvals; - - Chinese statistics and policy; - - NVDA filings; - - ai-timeline. - - It uses the same model endpoint, with no server changes. -2. **Stages 1 and 2** join the prototype. Stage 0's card feeds them. -3. **In the server, behind a base setting** (old path by default), in order: card and time (0, 4), normalisation (3), mentions and statements (1, 2), batch identity (7). -4. **The decision queues and their interface** come last. - -Each step lands only if every bench holds within its variance, and the audit metrics below improve. - -## Measurement - -- **Recall benches.** Every stage is measured on the five recall benches (`recall.mjs --corpus`, `lease_bench.py`), the identity bench over shuffled orders, and the governance bench. -- **Audit metrics** on the resulting bases: - - distinct predicates per fact, and the unbound share; - - description entities and names-only entities; - - aliases that fail the structural checks; - - entities split across type labels; - - facts without a start whose evidence states a date; - - statements whose ends do not appear in their evidence. -- **Normalisation bench.** Gold (phrase, context) → ontology key, reversed, new, or none. It measures binding precision, binding rate and direction accuracy, reported by frequency band. -- **Precision.** Precision needs sampled human labels. An LLM judge is used only after its agreement with those labels is known. -- **Cost.** Tokens and latency per document per stage, against today's single call. - -## Open questions - -- **Chinese coreference and time.** The mature open tools are English-first. Whether stage 1 coreference and stage 4 normalisation for Chinese are LLM calls or rule libraries is measured on the Chinese benches. -- **Atomic facts.** ATOM's decontextualised atomic facts raised fact recall by 31% and cost 9% precision ([paper](https://arxiv.org/html/2510.22590v2)). Whether stage 2 needs that layer is measured, not assumed. -- **XBRL.** Figures in SEC filings exist as structured facts; whether filings bypass stage 2 for them is a separate record. -- **Card cost on long documents.** It is measured on the 136-chunk earnings release before a section-card threshold is chosen. diff --git a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md new file mode 100644 index 000000000..cbf28f6a1 --- /dev/null +++ b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md @@ -0,0 +1,123 @@ +# 0044 · The ontology is a view over what documents say + +- **Status**: Proposed 2026-09-16 · replaces the staged-reading draft of this record (skim card, graded mentions, statements bound at write time), which the prototype below did not bear out · nothing built in the product · prototype scripts and measurements from 2026-09-15 are summarised in [What the prototype measured](#what-the-prototype-measured) +- **Written**: 2026-09-16 (conventions in the [README](README.md)) +- **Related**: [0022](0022-an-unknown-date-is-not-an-open-one.md) put a document's date in `attested_at` beside the world and record axes; [0025](0025-governance-reads-the-ledger-before-it-decides.md) and [0027](0027-an-automatic-merge-is-gated-by-what-it-can-undo.md) put agent decisions through a gate that weighs what they can undo; [0041](0041-a-name-is-a-claim-about-an-entity.md) made names facts and identity a matter of evidence; [0043](0043-every-review-queue-is-governed.md) sent every review queue through the governor; #714 found upload time used as the document date in extraction. + +> The introduction to *The Eminem Show* says the album won the Grammy for Best Rap Album. Given the knowledge base's properties, the extractor records that the album's genre is Best Rap Album, and that it received Best Rap Album as an award. Asked to write what the introduction says in its own words, it records that the album won Best Rap Album. + +## Context + +Today extraction binds each fact to a property of the knowledge base's ontology at the moment it is written. The ontology arrives in the prompt, whole or trimmed, and a fact that finds no property is kept as an unbound proposal. The audit recorded in the first draft of this record found 59% of facts unbound on the timeline corpus, 1,093 distinct predicates, and a fifth of entities named by descriptions. The first draft answered with more stages around the same act of binding. A prototype built on 2026-09-15 tested the act of binding itself, and found that the errors live there. + +## What the prototype measured + +Setup: DeepSeek-V4-Flash with thinking off for extraction and agents; DeepSeek-V4-Pro as judge, calibrated against 24 hand-labelled facts (it agreed on 21: 8 of 9 correct facts accepted, 2 of 15 wrong ones passed). Corpora: 100 documents sampled from the Re-DocRED test set (Wikipedia introductions, 3,462 gold triples, 95 Wikidata properties), and the FDA, statistics-bulletin and SEC-filing corpora of `scripts/bench/recall.mjs`. Counts are single runs unless stated; runs of the same configuration varied by about two F1 points. + +**1. Statements written in the document's own words are faithful.** Of 333 open statements from two extraction prompts, the judge found none that the document does not state, and 2% worded wrongly. + +| On 10 Re-DocRED documents | Statements | Stated | Stated but worded wrongly | Not stated | +|---|---|---|---|---| +| Open statements, full extraction prompt | 192 | 97.9% | 2.1% | 0 | +| Open statements, compact one-call prompt | 141 | 98.6% | 1.4% | 0 | +| Facts bound at write time, property names only | 78 | 74.4% | 16.7% | 9.0% | +| Facts bound at write time, with definitions | 114 | 79.8% | 15.8% | 4.4% | + +The bound facts that are not stated are induced by the list of properties: an award written as a genre, a county given a notable work, a seventeenth-century king given a modern citizenship. + +**2. Binding after the fact loses more than it keeps.** Aligning open statements to an existing ontology bound 4 of 113 to 567 phrase groups per corpus against the imported schema.org properties (FDA, statistics, SEC), and scored F1 9.1 on Re-DocRED against its own 95 properties, where extracting with the properties in the prompt scored 21.8. The loss comes from implicit facts ("a 1952 British film" has a country of origin) that the open statements do not state as statements. + +**3. Gold overlap understates precision.** Re-DocRED still omits many true facts. Strict precision against gold was 37.8% for the sliced extraction; the calibrated judge accepted 76.1%. A hand review of 30 unmatched facts found 9 correct, 5 with a near-synonymous property, 6 debatable and 10 wrong. Comparisons below hold under all three measures. + +**4. A large ontology can be sliced per document, if its structure is complete.** With the 95 benchmark properties hidden among 872 schema.org properties, each document got a slice of about 133 properties that contained 86.6% of the relations its gold triples use. The slice was the union of properties whose domain and range meet the classes of the document's entities (34.0% alone), properties nearest to its statements' phrases (28.9% alone), and every property with no declared domain (the most frequent ones, country and located-in, have none). Without the class hierarchy and the Wikidata–schema.org equivalences, the structural part found 10%. Extraction with the slice scored F1 18.3 against 21.8 with the 95 properties alone; merging the 49 schema.org properties Wikidata declares equivalent to benchmark properties raised it to 20.2 and restored precision (34.3% → 37.8%, judged 71.2% → 75.3%). + +**5. Open and bound extraction complement each other when they run independently.** Unioning the open statements with the bound facts linked 46.8% of gold entity pairs (open alone 38.4%, bound alone 34.5%) with F1 22.5. Giving the bound pass the open statements as context lowered recall to 42.9%. + +**6. An agent can correct after extraction.** An errata agent reviewed all facts of 99 documents with tools to read the document, look up a property's definition, constraints, broader properties and inverse, and retract, revise or add. Judged precision rose from 76.1% to 89.5% and strict precision from 37.8% to 47.1%; strict recall moved from 13.8% to 13.6%. It retracted 278 facts, revised 42 and added 49; the judge's count of correct facts fell from 871 to 814, so about a quarter of what it removed was right. Agent-written usage notes learned from half the documents and given to extraction on the other half helped no more than a generic caution of the same length. + +**7. Cost.** For a 165-word document the full prototype spent about 23,000 prompt and 7,200 completion tokens. A compact pipeline (entity scan, cached type mapping, one extraction call, errata only for flagged facts) spent 3,449 and 753, at F1 23.5 against 28.4 for the full prototype with errata on the same 10 documents. Mapping entity type words to classes is paid once per distinct word across a knowledge base: the second run over the same documents met 5 new words instead of 54. + +**8. Operational ontology platforms do not induce ontologies from text.** Their object types, links and actions are designed per use case by builders and backed by curated datasets; language models extract into a table a builder has already shaped, and edits are replayed over the backing data. + +## Decisions + +### 1. Three layers + +- **The open graph** holds what documents say: entities with their names and type words, statements whose relation is the document's phrase, with their participants, qualifiers, time mentions and quotes. It needs no ontology and is kept whatever the ontology becomes. +- **The ontology** holds object types, link types and properties, each with a definition, examples and regression cases, plus constraints and actions. It is small, shaped by the questions the knowledge base must answer, and versioned on the recorded axis. +- **The typed graph** holds facts expressed in the ontology. It is computed from the open graph and marked with the ontology version it was computed under. + +The word ontology in this project means the second layer only. + +### 2. Extraction writes the open graph and nothing else + +One call per chunk, a compact contract (arrays, short keys, no repeated names), no ontology in the prompt. The extractor reports entities with a type word, statements in the text's words, figures and titles as attributes of the thing they describe, time mentions as they are written, and the other names a document gives. Code checks that names and quotes occur in the chunk. + +### 3. The typed graph is materialised from the open graph + +A binding is decided once for a signature: the phrase, the subject's classes and the object's classes. The decision (a property and a direction, or none) is cached with its confidence and the ontology version, and reused wherever the signature recurs, so its cost grows with the number of distinct phrasings rather than documents. Facts a reader draws without the text stating them (a place's country from its region, a film's country from its nationality adjective) come from derivation rules over the typed graph and are marked derived. When the ontology changes, only facts under changed signatures are recomputed. A property missing from the ontology does not stop the statement: it stays in the open graph and becomes evidence for a proposal. + +### 4. The ontology is proposed by an agent and approved by people + +An ontology agent reads the open graph and a set of competency questions, and proposes object types, link types and properties with definitions, examples and the signatures they would bind. People approve through actions. Each approved element carries regression cases drawn from the open graph; changing a definition reruns them. The ontology is judged by whether the competency questions can be answered correctly. Structure the slice depends on (class hierarchy, equivalences, domains, ranges) is part of approval, and duplicate properties are merged as part of governance. + +### 5. Time is resolved the way identity is + +- A time mention is a fact with its quote, attached to the statements it dates. +- Extraction carries a document time context from chunk to chunk: the document's own date, taken from its content or a source that dates it (never the time it was uploaded, #714), the anchors its narrative sets, and the calendars it defines (a fiscal year, a reporting period). +- The model returns an interpretation, not a date: absolute, or an anchor with an offset; a granularity; a point, an interval or an as-of. Code computes the interval. +- A mention whose anchor is unknown keeps its words and waits; a later anchor, in the document or in another, triggers recomputation. +- Three times stay apart: when a statement holds (valid time), when the document observed it (its own date), and when the ledger learned it (recorded time). + +### 6. Identity across documents is decided on evidence + +Every document's entities, merged within the document on name and stated aliases, are profiled by names, classes, attributes, neighbours and time span. Candidates come from name facts and name vectors across scripts. Deterministic evidence scores each candidate pair and settles what it can: agreeing names and neighbours raise the score, and a conflicting type, attribute or lifespan is a cannot-link. Only undecided pairs go to the adjudicator, which sees two profiles rather than two documents. Clustering respects cannot-links, so A≈B and B≈C never merge A and C against evidence. New evidence re-evaluates earlier merges, and a merge or a split is an action that can be reverted. A rename keeps one entity with names valid at different times; a role such as a company's chief executive is not an entity. + +### 7. An errata agent reviews the typed graph after extraction + +It reviews facts flagged by structure (a subject or object outside the property's declared kinds, a name not found in the document, a date property without a date) before sampling the rest, checks every fact it is given before adding any, works through a JSON action protocol with a per-document budget, and records each retraction or revision as an action with the document's words as evidence. Its measure is precision gained against correct facts removed. + +### 8. Lexicons remain governed data + +Alias tables, bindings, cannot-link lists and gazetteers are data with provenance, reviewed like facts, never lists in code. + +## Not doing + +- **Binding to the ontology at write time.** It produced the extraction errors measured above and ties every document to one version of the ontology. +- **Putting a general ontology in the prompt.** A knowledge base's ontology grows with use; the slice depends on the document, and binding happens on signatures. +- **Learning usage notes from part of a corpus to re-extract the rest.** Measured no better than a generic caution. +- **Dates computed by the model, or upload time used as a document date.** +- **Word lists in code** for names, suffixes, relation shapes or time expressions. + +## Measurement + +Every cut reports on at least three domains, twice per configuration, with the judge's calibration stated. + +| What | Bench | Numbers | +|---|---|---| +| Open graph | Re-DocRED (100 documents) and the FDA, statistics and SEC corpora | statements not stated and worded wrongly (judge), entity-pair recall, tokens per document | +| Typed graph | Re-DocRED with its 95 properties as the approved ontology | strict, structural and judged precision, recall, F1; cost per document as the corpus grows | +| Ontology | FDA and statistics corpora with written competency questions | questions answered correctly; share of proposals people change | +| Time | `temporal.mjs`, the lease bench, SEC fiscal periods, statistics bulletins | normalised value and granularity, statement valid time, as-of answers | +| Identity across documents | `identity.mjs`; Linked-Re-DocRED documents that share non-location entities or namesakes of different types (GPL-3.0, kept outside the repository) | pairwise precision, recall, F1; wrong merges and missed merges apart; adjudicator calls per thousand entities | +| Errata | the typed-graph benches | precision gained, correct facts removed, tokens | + +Thresholds to pass before a cut lands: the open graph at or under 2% not-stated with entity-pair recall no lower than today's; the typed graph at or above the full prototype's F1 on the same Re-DocRED documents (22.5 on the 100-document sample before errata) at under a fifth of its tokens per document. + +## Cuts + +1. The open graph in the ledger: statements, time mentions and names with provenance on both clocks; the compact extraction contract behind a flag. +2. Signature bindings: a table of signature → property, direction, confidence, ontology version; materialisation and recomputation of the typed graph. +3. Time context and code resolution of time mentions (closes #714). +4. Identity evidence: profiles, deterministic scoring, cannot-links, constrained clustering; name vectors from 0041 cut 2 (its migration renumbered from 0060). +5. The ontology agent and competency questions; regression cases on definitions. +6. The errata agent on the typed graph through the gate. + +Today's extractor stays as it is until cut 2 passes its thresholds. + +## Open questions + +- How queries read a typed graph that is partly materialised, and when materialisation is eager. +- Whether derivation rules recover the implicit facts that write-time binding found (Re-DocRED's country and located-in relations are a fifth of its pairs). +- Competency questions for a new knowledge base that has none yet. +- How much of identity the deterministic evidence settles before the adjudicator is needed. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 933e4f505..330199085 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -67,6 +67,7 @@ The test for writing one: if someone (including us) looks at a piece of code in | 0041 | [A name is a claim about an entity](0041-a-name-is-a-claim-about-an-entity.md) | Cut 0 built (identity bench) · cut 1 implemented (#670): names are value facts on `known_as`, the extractor reports other names, a shared name goes to the adjudicator; forward/reverse F1 0.43/0.54 → 0.68/0.68 · cuts 2–4 (name vectors and neighbours, evidence decides, re-evaluation) not started | | 0042 | [The chat loop is a runner with hooks](0042-the-chat-loop-is-a-runner-with-hooks.md) | Implemented (#548) · the loop is rig's runner and every policy is a hook with a typed result · the wire stays `LlmClient` behind `RigModel` · a turn cannot end before a tool has run, `no_evidence_needed` is the exit for questions not about the base · RAG fallback only on a 400/422 to the first request with tools · an empty reply is asked again once · the skip rate is the model's (DeepSeek-V3 1–3 of 12, Qwen2.5-72B 0) and recorded, not prevented | | 0044 | [A document is read in stages](0044-a-document-is-read-in-stages.md) | Proposed · nothing built · extraction becomes stages with narrow contracts: a skim that writes a document card with evidenced fields, graded mentions where descriptions never become entities, statements in the text's words with ends from the inventory, normalisation once per distinct phrase with proposals through governance, time resolved by code, identity by evidence in batch · reviews become decisions that write back · today's extractor frozen until an offline prototype of stages 0 and 3 reports on five benches | +| 0044 | [The ontology is a view over what documents say](0044-the-ontology-is-a-view-over-what-documents-say.md) | Proposed · nothing built · replaces the staged-reading draft · three layers: extraction writes an open graph in the documents' words (0 of 333 statements unstated in the prototype, against 4–9% of facts bound at write time), a small ontology proposed by an agent and approved by people, and a typed graph computed from the open graph on cached signatures · time mentions resolved against a document time context by code · identity across documents on deterministic evidence before the adjudicator · an errata agent reviews the typed graph | ## Not a decision record From af8f0a3afa423a8b5fc5227c9424dde48129be52 Mon Sep 17 00:00:00 2001 From: WaylandYang Date: Wed, 16 Sep 2026 01:31:55 +0800 Subject: [PATCH 5/7] 0044 reports judged precision and gold recall instead of F1 Co-Authored-By: Claude Opus 5 Signed-off-by: WaylandYang --- ...ology-is-a-view-over-what-documents-say.md | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md index cbf28f6a1..1069f4c1b 100644 --- a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md +++ b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md @@ -12,7 +12,9 @@ Today extraction binds each fact to a property of the knowledge base's ontology ## What the prototype measured -Setup: DeepSeek-V4-Flash with thinking off for extraction and agents; DeepSeek-V4-Pro as judge, calibrated against 24 hand-labelled facts (it agreed on 21: 8 of 9 correct facts accepted, 2 of 15 wrong ones passed). Corpora: 100 documents sampled from the Re-DocRED test set (Wikipedia introductions, 3,462 gold triples, 95 Wikidata properties), and the FDA, statistics-bulletin and SEC-filing corpora of `scripts/bench/recall.mjs`. Counts are single runs unless stated; runs of the same configuration varied by about two F1 points. +Setup: DeepSeek-V4-Flash with thinking off for extraction and agents; DeepSeek-V4-Pro as judge, calibrated against 24 hand-labelled facts (it agreed on 21: 8 of 9 correct facts accepted, 2 of 15 wrong ones passed). Corpora: 100 documents sampled from the Re-DocRED test set (Wikipedia introductions, 3,462 gold triples, 95 Wikidata properties), and the FDA, statistics-bulletin and SEC-filing corpora of `scripts/bench/recall.mjs`. Counts are single runs unless stated; three runs of one configuration on 50 documents varied by four points of judged precision and one point of gold recall. + +No F1 is reported. Precision against Re-DocRED's gold is understated by how many true facts the gold omits (item 3), and by a different amount for each approach, so a harmonic mean of it with recall compares nothing. Precision is the calibrated judge's; recall is the share of gold facts recovered, which is sound because every gold fact is true. **1. Statements written in the document's own words are faithful.** Of 333 open statements from two extraction prompts, the judge found none that the document does not state, and 2% worded wrongly. @@ -25,17 +27,17 @@ Setup: DeepSeek-V4-Flash with thinking off for extraction and agents; DeepSeek-V The bound facts that are not stated are induced by the list of properties: an award written as a genre, a county given a notable work, a seventeenth-century king given a modern citizenship. -**2. Binding after the fact loses more than it keeps.** Aligning open statements to an existing ontology bound 4 of 113 to 567 phrase groups per corpus against the imported schema.org properties (FDA, statistics, SEC), and scored F1 9.1 on Re-DocRED against its own 95 properties, where extracting with the properties in the prompt scored 21.8. The loss comes from implicit facts ("a 1952 British film" has a country of origin) that the open statements do not state as statements. +**2. Binding after the fact loses more than it keeps.** Aligning open statements to an existing ontology bound 4 of 113 to 567 phrase groups per corpus against the imported schema.org properties (FDA, statistics, SEC), and recovered 5.6% of Re-DocRED's gold facts against its own 95 properties, where extracting with the properties in the prompt recovered 15.3%. The loss comes from implicit facts ("a 1952 British film" has a country of origin) that the open statements do not state as statements. -**3. Gold overlap understates precision.** Re-DocRED still omits many true facts. Strict precision against gold was 37.8% for the sliced extraction; the calibrated judge accepted 76.1%. A hand review of 30 unmatched facts found 9 correct, 5 with a near-synonymous property, 6 debatable and 10 wrong. Comparisons below hold under all three measures. +**3. Gold overlap understates precision.** Re-DocRED still omits many true facts. Strict precision against gold was 37.8% for the sliced extraction; the calibrated judge accepted 76.1%. A hand review of 30 unmatched facts found 9 correct, 5 with a near-synonymous property, 6 debatable and 10 wrong. The orderings below hold for strict and judged precision alike; the judged figures are the ones quoted. -**4. A large ontology can be sliced per document, if its structure is complete.** With the 95 benchmark properties hidden among 872 schema.org properties, each document got a slice of about 133 properties that contained 86.6% of the relations its gold triples use. The slice was the union of properties whose domain and range meet the classes of the document's entities (34.0% alone), properties nearest to its statements' phrases (28.9% alone), and every property with no declared domain (the most frequent ones, country and located-in, have none). Without the class hierarchy and the Wikidata–schema.org equivalences, the structural part found 10%. Extraction with the slice scored F1 18.3 against 21.8 with the 95 properties alone; merging the 49 schema.org properties Wikidata declares equivalent to benchmark properties raised it to 20.2 and restored precision (34.3% → 37.8%, judged 71.2% → 75.3%). +**4. A large ontology can be sliced per document, if its structure is complete.** With the 95 benchmark properties hidden among 872 schema.org properties, each document got a slice of about 133 properties that contained 86.6% of the relations its gold triples use. The slice was the union of properties whose domain and range meet the classes of the document's entities (34.0% alone), properties nearest to its statements' phrases (28.9% alone), and every property with no declared domain (the most frequent ones, country and located-in, have none). Without the class hierarchy and the Wikidata–schema.org equivalences, the structural part found 10%. Extraction with the slice reached judged precision 71.2% and gold recall 12.4%, against 77.8% and 15.3% with the 95 properties alone; merging the 49 schema.org properties Wikidata declares equivalent to benchmark properties raised them to 75.3% and 13.8%. -**5. Open and bound extraction complement each other when they run independently.** Unioning the open statements with the bound facts linked 46.8% of gold entity pairs (open alone 38.4%, bound alone 34.5%) with F1 22.5. Giving the bound pass the open statements as context lowered recall to 42.9%. +**5. Open and bound extraction complement each other when they run independently.** Unioning the open statements with the bound facts linked 46.8% of gold entity pairs (open alone 38.4%, bound alone 34.5%) and recovered 16.0% of gold facts. Giving the bound pass the open statements as context lowered recall to 42.9%. -**6. An agent can correct after extraction.** An errata agent reviewed all facts of 99 documents with tools to read the document, look up a property's definition, constraints, broader properties and inverse, and retract, revise or add. Judged precision rose from 76.1% to 89.5% and strict precision from 37.8% to 47.1%; strict recall moved from 13.8% to 13.6%. It retracted 278 facts, revised 42 and added 49; the judge's count of correct facts fell from 871 to 814, so about a quarter of what it removed was right. Agent-written usage notes learned from half the documents and given to extraction on the other half helped no more than a generic caution of the same length. +**6. An agent can correct after extraction.** An errata agent reviewed all facts of 99 documents with tools to read the document, look up a property's definition, constraints, broader properties and inverse, and retract, revise or add. Judged precision rose from 76.1% to 89.5%; gold recall moved from 13.8% to 13.6%. It retracted 278 facts, revised 42 and added 49; the judge's count of correct facts fell from 871 to 814, so about a quarter of what it removed was right. Agent-written usage notes learned from half the documents and given to extraction on the other half helped no more than a generic caution of the same length. -**7. Cost.** For a 165-word document the full prototype spent about 23,000 prompt and 7,200 completion tokens. A compact pipeline (entity scan, cached type mapping, one extraction call, errata only for flagged facts) spent 3,449 and 753, at F1 23.5 against 28.4 for the full prototype with errata on the same 10 documents. Mapping entity type words to classes is paid once per distinct word across a knowledge base: the second run over the same documents met 5 new words instead of 54. +**7. Cost.** For a 165-word document the full prototype spent about 23,000 prompt and 7,200 completion tokens. A compact pipeline (entity scan, cached type mapping, one extraction call, errata only for flagged facts) spent 3,449 and 753. On the same 10 documents it reached judged precision 80.0% and gold recall 15.1%, against 85.9% and 18.9% for the full prototype's bound facts after errata. Mapping entity type words to classes is paid once per distinct word across a knowledge base: the second run over the same documents met 5 new words instead of 54. **8. Operational ontology platforms do not induce ontologies from text.** Their object types, links and actions are designed per use case by builders and backed by curated datasets; language models extract into a table a builder has already shaped, and edits are replayed over the backing data. @@ -96,13 +98,13 @@ Every cut reports on at least three domains, twice per configuration, with the j | What | Bench | Numbers | |---|---|---| | Open graph | Re-DocRED (100 documents) and the FDA, statistics and SEC corpora | statements not stated and worded wrongly (judge), entity-pair recall, tokens per document | -| Typed graph | Re-DocRED with its 95 properties as the approved ontology | strict, structural and judged precision, recall, F1; cost per document as the corpus grows | +| Typed graph | Re-DocRED with its 95 properties as the approved ontology | judged precision with the judge's agreement on a hand-labelled set; gold recall, split into facts whose two ends share a sentence and facts that need more than one; cost per document as the corpus grows. Strict precision, recall and F1 only beside published results | | Ontology | FDA and statistics corpora with written competency questions | questions answered correctly; share of proposals people change | | Time | `temporal.mjs`, the lease bench, SEC fiscal periods, statistics bulletins | normalised value and granularity, statement valid time, as-of answers | -| Identity across documents | `identity.mjs`; Linked-Re-DocRED documents that share non-location entities or namesakes of different types (GPL-3.0, kept outside the repository) | pairwise precision, recall, F1; wrong merges and missed merges apart; adjudicator calls per thousand entities | +| Identity across documents | `identity.mjs`; Linked-Re-DocRED documents that share non-location entities or namesakes of different types (GPL-3.0, kept outside the repository) | pairwise precision and recall, with wrong merges and missed merges apart; adjudicator calls per thousand entities | | Errata | the typed-graph benches | precision gained, correct facts removed, tokens | -Thresholds to pass before a cut lands: the open graph at or under 2% not-stated with entity-pair recall no lower than today's; the typed graph at or above the full prototype's F1 on the same Re-DocRED documents (22.5 on the 100-document sample before errata) at under a fifth of its tokens per document. +Thresholds to pass before a cut lands: the open graph at or under 2% not-stated with entity-pair recall no lower than today's; the typed graph at or above the full prototype on the same Re-DocRED documents in judged precision (75.3% on the 100-document sample before errata) and in gold recall of facts whose ends share a sentence, each reported over two runs, at under a fifth of its tokens per document. Recall of facts that need more than one sentence is reported and waits for derivation rules before it becomes a threshold. ## Cuts From 023289dc9df6c3c4fd50e450e3071001e2462ab0 Mon Sep 17 00:00:00 2001 From: WaylandYang Date: Wed, 16 Sep 2026 13:47:48 +0800 Subject: [PATCH 6/7] 0044 makes alignment produce bindings and implication rules on a workbench Co-Authored-By: Claude Fable 5.1 Signed-off-by: WaylandYang --- ...ntology-is-a-view-over-what-documents-say.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md index 1069f4c1b..de9a70b31 100644 --- a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md +++ b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md @@ -55,13 +55,20 @@ The word ontology in this project means the second layer only. One call per chunk, a compact contract (arrays, short keys, no repeated names), no ontology in the prompt. The extractor reports entities with a type word, statements in the text's words, figures and titles as attributes of the thing they describe, time mentions as they are written, and the other names a document gives. Code checks that names and quotes occur in the chunk. -### 3. The typed graph is materialised from the open graph +### 3. Alignment produces the typed graph: bindings and implication rules -A binding is decided once for a signature: the phrase, the subject's classes and the object's classes. The decision (a property and a direction, or none) is cached with its confidence and the ontology version, and reused wherever the signature recurs, so its cost grows with the number of distinct phrasings rather than documents. Facts a reader draws without the text stating them (a place's country from its region, a film's country from its nationality adjective) come from derivation rules over the typed graph and are marked derived. When the ontology changes, only facts under changed signatures are recomputed. A property missing from the ontology does not stop the statement: it stays in the open graph and becomes evidence for a proposal. +Alignment is an operation of the review workbench over the open graph, decided per signature (the phrase, the subject's classes, the object's classes), never per fact. It produces two things: -### 4. The ontology is proposed by an agent and approved by people +- **A binding**: this signature is this property in this direction, or none. Cached with its confidence and the ontology version, reused wherever the signature recurs, so the cost grows with distinct phrasings rather than documents. +- **An implication rule**: a statement of this shape implies a fact of that property, with the value taken from a side of the statement or read from the object phrase by a stated reading (the country a nationality adjective names, the year a phrase gives). The aligner proposes the rule, the workbench approves it, and code executes it; a reading is applied once per distinct phrase and cached. Facts the rules produce are marked implied. -An ontology agent reads the open graph and a set of competency questions, and proposes object types, link types and properties with definitions, examples and the signatures they would bind. People approve through actions. Each approved element carries regression cases drawn from the open graph; changing a definition reruns them. The ontology is judged by whether the competency questions can be answered correctly. Structure the slice depends on (class hierarchy, equivalences, domains, ranges) is part of approval, and duplicate properties are merged as part of governance. +This is how the facts a reader draws without the text stating them (a place's country from its region, a film's country from "British film") reach the typed graph without a second extraction pass bound to the ontology. Whether rules recover as much as the bound pass recovered in the prototype (16.0% of gold facts, 44.6% of inferred reference facts, against 5.6% and 30.4% for binding alone) is measured before cut 2 lands. + +When the ontology changes, only facts under changed signatures and rules are recomputed. A signature with no property stays in the open graph, loses nothing, and counts toward the workbench's suggestions. + +### 4. The ontology is built on a workbench from three sources + +The ontology page becomes a workbench. Its elements come from three sources: **suggestions from the open graph** (the most frequent unbound signatures, the type words in use, and an ontology agent that reads them against competency questions and proposes object types, link types, properties and rules with definitions, examples and the signatures they would bind); **an imported file** (a pack of 0008, schema.org, an OWL or JSON-LD file); and **online editing**. People approve through actions; every approved element carries regression cases drawn from the open graph, and changing a definition reruns them. The ontology is judged by whether the competency questions can be answered correctly. Structure the slice and the rules depend on (class hierarchy, equivalences, domains, ranges) is part of approval, and duplicate properties are merged as part of governance. ### 5. Time is resolved the way identity is @@ -109,7 +116,7 @@ Thresholds to pass before a cut lands: the open graph at or under 2% not-stated ## Cuts 1. The open graph in the ledger: statements, time mentions and names with provenance on both clocks; the compact extraction contract behind a flag. -2. Signature bindings: a table of signature → property, direction, confidence, ontology version; materialisation and recomputation of the typed graph. +2. Alignment: a table of signature → property, direction, confidence, ontology version; implication rules and their cached readings; materialisation and recomputation of the typed graph. 3. Time context and code resolution of time mentions (closes #714). 4. Identity evidence: profiles, deterministic scoring, cannot-links, constrained clustering; name vectors from 0041 cut 2 (its migration renumbered from 0060). 5. The ontology agent and competency questions; regression cases on definitions. From 6dd3b248ba6528f730322ae6c13ff2f76a4eb6b7 Mon Sep 17 00:00:00 2001 From: WaylandYang Date: Wed, 16 Sep 2026 16:01:58 +0800 Subject: [PATCH 7/7] 0044 says alignment reads the source Co-Authored-By: Claude Fable 5.1 Signed-off-by: WaylandYang --- .../0044-the-ontology-is-a-view-over-what-documents-say.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md index de9a70b31..b91e3b33e 100644 --- a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md +++ b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md @@ -62,7 +62,9 @@ Alignment is an operation of the review workbench over the open graph, decided p - **A binding**: this signature is this property in this direction, or none. Cached with its confidence and the ontology version, reused wherever the signature recurs, so the cost grows with distinct phrasings rather than documents. - **An implication rule**: a statement of this shape implies a fact of that property, with the value taken from a side of the statement or read from the object phrase by a stated reading (the country a nationality adjective names, the year a phrase gives). The aligner proposes the rule, the workbench approves it, and code executes it; a reading is applied once per distinct phrase and cached. Facts the rules produce are marked implied. -This is how the facts a reader draws without the text stating them (a place's country from its region, a film's country from "British film") reach the typed graph without a second extraction pass bound to the ontology. Whether rules recover as much as the bound pass recovered in the prototype (16.0% of gold facts, 44.6% of inferred reference facts, against 5.6% and 30.4% for binding alone) is measured before cut 2 lands. +Alignment reads the source. Deciding a signature and applying a reading both see the chunk the statements came from, so a statement whose object is a phrase that names a thing ("located in the Piedmont region of the Commonwealth of Virginia") binds to the property and to the thing, as a reader would. Alignment is a typed reading of the text organised by signature, with the open graph as its index and the text as its evidence; it differs from binding at write time in that it never writes a fact without an open statement or a rule behind it, and never re-extracts a document when the ontology changes. + +This is how the facts a reader draws without the text stating them (a place's country from its region, a film's country from "British film") reach the typed graph without a second extraction pass bound to the ontology. The first prototype of alignment read only the statements: on 30 Re-DocRED documents it recovered 6.5% of gold facts by bindings and 10–12% with rules, against 15.5% for the bound pass, and the gap sat where the aligner had not read the text. Until alignment reaches the bound pass on two clean runs, a bound pass with the ontology slice remains an option for an approved ontology, and its facts must attach to an open statement or be marked implied. When the ontology changes, only facts under changed signatures and rules are recomputed. A signature with no property stays in the open graph, loses nothing, and counts toward the workbench's suggestions.