Temporal question answering has no benchmark to report

[WaylandYang]

The one number this project can quote is Ontology2SQL on BIRD Mini-Dev. Nothing measures the thing it is actually built around: answering a question as of a date, and getting the answer right when the fact changed.

The existing memory benchmarks do not test this. LongMemEval and DMR measure conversational recall, which is Zep's home ground, and neither asks "who ran this project in March 2023" over a corpus where the answer changed twice. Reporting a mid-table score on someone else's benchmark says less than publishing the benchmark that is missing.

What it needs: a corpus where facts change on known dates, questions whose answers depend on the as-of point, and a scoring script anyone can re-run. The e2e setup already builds this kind of data by hand for every temporal fix — the Phoenix succession, the salary correction, the merge-then-reconcile cases. Turning that into a fixed, versioned set makes the regressions visible too, which is worth more than the headline figure.

[WaylandYang]

Where the numbers stand after the first week of running it (dev @ d2bfb62, DeepSeek-V3 + bge-m3, 37 questions, three waves; two questions pinned as known gaps and kept out of the main score).

The ledger, asked directly

configuration	main	world axis	record axis	known gaps
axioms declared before ingest	35 / 35	21 / 21	14 / 14	0 / 2
ontology grown, nothing declared	28 / 35	15 / 21	13 / 14	1 / 2
ontology grown, declared afterwards and reconciled (#354)	34 / 35	21 / 21	13 / 14	1 / 2

The gap between the first two rows was #341; the third row is that issue closed — the path a new deployment walks, with a person accepting the declaration after the fact. The record axis is at full marks on a declared base: correction rewind (#311), merge rewind (#337) and deletion tombstones (#268) all hold on the real ingestion path.

The same questions, asked through Chat — 28 / 35. This row measures whether the agent can ask the ledger, and it moved from 24 to 28 with #349 (the model is shown attribute values) and #350 (the tools take as_of). Of the seven misses, three are right answers the substring scorer cannot read (a forbidden name inside a negation), two are the model emitting its tool call as text, two are "before the memo arrived" questions where the model asks for the date instead of reading it from changes — and where #351 would have stopped it anyway. Run-to-run variance on this row is about ±2.

What the benchmark found and filed: #341 (closed by #354), #345, #347 (closed by #350), #348 (closed by #349), #351, #352.

Method, question sheet and the reasons behind every number: scripts/bench/README.md. Reproduce with node scripts/bench/temporal.mjs --label X [--no-declare [--reconcile]] [--chat].