A benchmark that asks the question this project is built around

[WaylandYang]

Closes #306.

The only number this repository could quote was Ontology2SQL on BIRD Mini-Dev. Nothing measured the thing it is actually built around — answering as of a date, and being right when the fact changed. scripts/bench/temporal.mjs measures it, over a fixed corpus and a fixed question sheet that anyone can re-run.

Two axes, which is the part nobody else has

axis	parameter	asks
world	at	what the world looked like then
record	as_of	what we thought the world looked like then

The same entity on the same date can have different right answers on the two axes, and aurora-lead-record-wave1-at-2024-08 is that question: the world axis asks about August 2024 (Li Si had taken over), the record axis stops before the handover memo arrived (we only knew Zhang San). Answering Li Si is answering the past with today's knowledge.

The record axis is not manufactured by moving clocks. The corpus is ingested in two waves; the timestamp taken after wave one is the record-axis boundary, and wave two brings the handover, a payroll correction and a document deletion. Every as_of: "wave-1" question asks about a moment the product actually passed through.

First results — 2026-09-05, dev @ 1dd917b, DeepSeek-V3 + bge-m3

configuration	ledger	world	record
uniqueness axioms declared	25 / 25	17 / 17	8 / 8
ontology left to grow itself (--no-declare)	15 / 25	8 / 17	7 / 8

The gap is the finding, and it is entirely on the world axis. Automatic closure runs off functional / inverse_functional, and those are never inferred — deliberately, since they drive automatic rewrites of the ledger. On a base where nobody declared them, a succession is stored as two open facts:

Zhang San | leads | Project Aurora | from 2023-01-10 | (no end) | live
Li Si     | leads | Project Aurora | from 2024-07-05 | (no end) | live

Two people running one project at the same time. Ask who ran it in June 2023 and both come back — on the question this product exists to answer. That is the shape a new deployment has out of the box, so it is filed as its own issue rather than buried in a benchmark table.

The record axis barely moves (7/8 vs 8/8) because it depends on no declaration at all: recorded_at and invalidated_at are written as the rows are written.

Two more things the run surfaced, recorded rather than smoothed over

• at omitted means "no world filter", not "now" — while as_of omitted does mean now. The two axes have different defaults, and aurora-lead-unfiltered pins that behaviour so a later change has to notice it.
• One salary value was dropped by extraction (extraction_drops, malformed_item): with number declared, the run where the model wrote "28000 CNY" failed validation. Refusing a dirty value is the deliberate choice; the benchmark just records that it happened.

What is in the sheet

25 questions over one fictional company — successions, a corrected salary, an employment that predates the document stating it, a retracted document, and three questions whose right answer is nothing (before a project started, before someone was hired, after a fact was retracted). Reading an open interval as "always was" is the easiest bitemporal mistake to make, so it gets its own questions.

Scoring does not look at predicate names: an ontology may call it works_for or employed_by, and this bench measures time, not vocabulary. absent (extraction never produced the entity) is its own bucket rather than a temporal miss, as in the type-resolution bench next door. --chat runs the same sheet through the agent for the product-level number; the ledger run is the regression net.

Verified by running it: the numbers above are from three full runs against a scratch deployment (its own database, its own data directory, a real model), not from a dry run.

🤖 Generated with Claude Code