A timeline is placed whole, whatever order its values arrive in

[WaylandYang]

The temporal engine now keeps a single-valued state's timeline whole, whatever order its values arrive in. This PR also carries the two #681 decisions that live in the engine: §1, start-less values, and §4, deadlines stated relative to an event. It revises 0022, whose status line and index row are updated, and adds migration 0057.

Why

Amendments are extracted in parallel, and historical filings are uploaded in any order, so a new value often lands in the middle of a history. The engine compared a new value only with the open rows, which is at most the latest value. On the Blackbaud HQ lease chain on EDGAR, three extended deadlines all closed at 2020-06-08, so on 2020-05-01 the lease had three deadlines.

Two review rounds shaped the design.

• First version: patched that one case. Review found six more ways it broke.
• Second version: tidied the timeline in rounds. It recognised an end the engine had drawn by one test: the end equals another row's start. Review found that this breaks as soon as the successor moves (a stated start arrives later, or earlier evidence turns up), and found four more problems:
  • a silent round cap;
  • residual order dependence;
  • a merge revert that deadlocked with extraction;
  • a revert that brought back a rejected value and dropped a person's interval edit.

This version removes the test instead of patching around it.

What changes

The row says who drew its end (migration 0057, facts.end_derived). The flag is true only on rows the temporal engine closed. It stays false on:

• an end the text states;
• a person's interval correction;
• a person's conflict decision;
• every row that existed before this migration. The migration does not guess who closed those.

A timeline is recomputed in one pass (temporal.rs, plan_timeline). All live rows of one (base, holder, predicate, side) are ordered by start, or, for a start-less row, by its earliest dated evidence.

• A row with a stated end is left alone.
• Every other row (open, or closed by the engine) ends where the next different value with confidence ≥ 0.75 begins, or stays open.
• A less confident successor in between goes to review and is skipped.

Rewriting an end never changes a row's start, value or confidence. So one pass is final, there are no rounds, and the result depends only on which rows exist. plan_timeline is a pure function with unit tests, one of which shuffles the input order.

Every change to a timeline recomputes it.

• A value arrives, or is observed again. Extraction and pending confirmation now reconcile a re-observed fact too, after writing its evidence: an earlier document can move a start-less row's anchor.
• A merge moves facts in.
• A merge is reverted.
• A person rejects a fact. reject_fact and the conflict decision reject_new go through temporal::retract.
• A document is deleted or restored.

Lock order. Reconciliation takes the timeline's advisory lock, then locks rows. A transaction that touches several timelines takes all their advisory locks first, in a fixed order (lock_timelines), and only then changes rows. The transactions that do this are merge revert, document deletion, document restore and retraction. Neither side ever holds a row lock while waiting for an advisory lock.

Merge revert moves back what the merge moved, with every row rewritten from it (resolution.rs), following supersedes. That includes a row the engine closed after the merge, a person's interval correction, and a rejected row, which stays rejected.

• It no longer restores and invalidates the merge's own corrections one by one. Both timelines are recomputed after the move, so an end that belonged to a value that went back to the source reopens.
• temporal_corrections is still recorded on the merge, for audit.

§1: a start-less row has a place (revision note in 0022). Its ordering date is the earliest date among its evidence documents whose own date is known (doc_time_source content or source); upload time and file mtime don't count.

• When it follows a predecessor: the predecessor ends ended-unknown, with valid_to NULL, precision 'unknown', and attested_to at that date. The reads therefore abut the two rows.
• Never: a document date is not written into valid_from or valid_to.

§4: a deadline relative to an event is kept. The model marks such a value relative: true; the server matches no wording.

• utopia_extract::attr_object_value stores it as written with "relative": true, and only when the value does not parse as a date. That function is now the tested glue the extraction loop calls.
• It closes the dated deadline before it, like any new value.
• RDF export: writes it as a plain xsd:string literal, plus utopia:relativeValue true on the statement. It is never an invalid xsd:date.
• Rule 8a now points at rule 10's flag.

Third review (8ff7362)

• A relation unique on both sides (functional and inverse-functional) puts each row on two timelines. The row now ends at the earlier of the two ends its sides draw, and such a predicate takes one lock, so a recompute can read the other side. Before, one side closed the row and a re-extraction reopened it from the other side.
• A start-less row whose text says it has ended no longer acts as a successor. Its evidence date shows it ended by then, not that it held then (Row::key).
• Merge revert keeps recording-axis replay intact. Rows on the merge ledger move back in place. A live row rewritten after the merge is superseded by a new row on the source (temporal::rehome_tx), and invalidated ones stay on the target.
• Document delete and restore re-read the cited facts after taking the locks, and lock any timelines that turned up late. They always recompute, and DATED_AT ignores deleted documents.
• Conflicts:
  • A rewrite carries the row's open conflicts to the new row, so a simultaneous or no_time pair is no longer withdrawn silently.
  • A pair a person kept is not asked again for rows descended from it.
  • A value with no time and a dated one become a no_time pair whichever arrives first; neither is closed.
• A document stating the end the engine drew turns that end into a stated one (temporal::state_derived_end, from the three reuse paths in insert_fact_inner).
• Evidence (graph::add_evidence) locks the fact row FOR SHARE and follows supersedes to the live row, so evidence never lands on a row that was just rewritten.
• Migration 0057 backfills end_derived on rows the old engine closed. All of these must hold:
  • the parent was open, with the same start and value;
  • the end equals another value's start on the same timeline;
  • there is no evidence the parent lacked;
  • no closed conflict decision, and no fact.close or fact.time_corrected audit on the parent.
• merge_entities now takes the timeline locks before it moves any row, the same way revert does.

Checks

• a_timeline_is_recomputed_from_the_rows_it_has (15 tests, the second review's reproductions):
  • a stated start that arrives later;
  • earlier evidence for a start-less successor;
  • a start-less value that arrives first;
  • a closed and an open value meeting in either order;
  • 70 low-confidence values followed by a confident one;
  • a merge of two interleaved 100-value timelines;
  • a merge and revert that restores both timelines exactly;
  • a moved value closed after the merge, then reverted;
  • revert keeps a person's correct_interval row;
  • revert keeps a rejected fact rejected;
  • rejecting a successor reopens its predecessor;
  • a person's stated end is never recomputed;
  • deleting and restoring a document moves the end it drew;
  • 10 trials each of 3 and 5 concurrent inserts;
  • 40 trials of a revert racing two arrivals: no error, no overlap.
• a_timeline_holds_whatever_the_order (8, from the first review): one assertion changed. A reverted value comes back as a rewritten row rather than the old row un-invalidated, so the test compares values, not ids.
• a_late_value_takes_its_place_in_history (5).
• Unit tests: plan_timeline (3), attr_object_value, and RDF a_relative_deadline_is_a_string_that_says_it_is_relative.
• the_engine_redraws_only_what_it_drew (11, the third review's reproductions): a relation unique on both sides stays put under re-extraction; an ended start-less value does not close the current one (two shapes); replay of the merge window is unchanged by revert; a deletion that waited for the lock still takes the rewritten row; evidence under 5-way concurrency (30 trials); a kept pair isn't asked again; an open conflict survives a rewrite; a stated end on an engine-closed row survives rejecting its successor; a deleted document no longer dates a row; a no-time value meets a dated one the same way in either order; the 0057 backfill marks only the engine's closure.
• an_undated_upload_is_not_a_document_date now expects the no_time pair (both open) instead of a closure.
• The reviewer's 23 third-round probes: 22 pass as written. The 23rd simulates a base without the backfill and is covered by the backfill test instead.
• Suites (fresh database migrated 0001→0057, no live server attached): cargo test -p utopia-store 259 passed, 0 failed; -p utopia-server 277 passed.
• Lint: cargo clippy --all-targets and fmt --check are clean.

Lease bench: run on the previous head, 98a3d20, before this rework: 7 and 8 of 18. Every live timeline traced there matched its inputs. The loss is extraction:

• 21 deadline values dropped as attr_datatype, because the model wrote "March 17, 2020" instead of ISO (A date the text spells out is read as a date #688);
• the 8th amendment's deadlines dropped as malformed_item;
• a landlord given the lease's date as its start.

On b2facea (the second rework) the lease bench scored 9 and 8 of 18, and every timeline matched its inputs; the remaining losses were extraction (written dates, fixed in #689).

Migration number: #684 took 0056 and is merged; this PR is 0057 and applies after it.

Fourth review, fixed in 7d79616

A fourth review round confirmed every fix from the third round. It ran the backfill on copies of three real bases: none of the flagged rows had a human trace, and reconcile_predicate afterwards only corrected overlaps the old engine had left. It also found problems in code the third round added. This branch's owning session had ended, so the fixes were made on top:

• Undoing the first merge of a chain left the source's facts on the chain's end. Revert only moved rows still held by the merge target. With S→T and then T→C, reverting S→T revived S but left S1 on C. Revert now follows merged_into from the target and moves rows held by any entity on that chain. Test: undoing_the_first_merge_of_a_chain_brings_its_facts_home.
• Deleting a document that cites thousands of timelines was slow and, at scale, impossible. One advisory lock per timeline: 8,000 timelines took 71 s, and at 20,000 the delete failed with out of shared memory. Locks are now two-level. A normal lock takes a shared predicate lock, then the exclusive timeline lock. Past 256 timelines, the transaction takes exclusive predicate locks instead of per-timeline locks. Predicate locks always come before timeline locks, each group sorted. The 20,000-timeline delete and restore now complete. Test: deleting_a_document_that_dates_many_timelines_still_works (REDRAW_BULK_N=20000 reproduces the review's scale).
• Undoing a merge dropped open conflicts on rehomed rows. rehome_tx now carries open conflicts to the new row before invalidating the old one. Test: a_revert_keeps_the_conflict_its_rows_carried.
• Restore took its locks in two rounds, so a transaction taking the same two locks in sorted order could deadlock it. The cited and revived timelines are now locked in one round.
• Deleting a row's earliest dated document left the reads overlapping. The engine's anchor moved to the next document, but attested_from (the read start of a start-less row) did not. Delete and restore now recompute attested_from from the remaining dated evidence. Test: deleting_the_first_document_moves_the_read_start_with_the_anchor.
• A reconcile pass recorded the same conflict pair again with old and new swapped. An open pair recorded in either order now counts as already recorded.

Not changed, noted:

• The 0057 backfill cannot tell an end the text stated from the engine's closure when the same chunk states both.
• A file-mtime document date can start a read earlier than the engine's anchor. That leaves an overlap with an open no_time conflict for a person.
• A document delete that touches thousands of timelines still recomputes all of them in one transaction: 8,000 took about 80 s, holding the predicate locks.

Checks on 7d79616 (fresh database, migrated 0001→0057):

• Suites: store, server and extract pass 592 tests with 0 failures. clippy --workspace --all-targets -D warnings is clean.
• Review probes (23): 22 pass; the 23rd simulated a base without the backfill and is superseded by the real-migration test.
• Attack tests (13): all pass, including chain revert, conflict carry and restore lock order.
• Stress: revert vs arrivals (120 trials); delete, reject, restore, merge, merge-vs-revert and correct vs arrivals; revert vs delete, evidence and reject (40 trials). 0 errors, 0 deadlocks, 0 overlaps.

🤖 Generated with Claude Code

[WaylandYang]

Back to draft after review; this needs a redesign before it can merge. The review found six problems, with reproductions:

1. Newest-first arrival still overlaps. A value that starts before all known history ignores the closed intervals after it. Arriving 9th, 7th, 5th, 6th leaves 6th [03-17, 06-08) over 7th [04-14, 06-08).
2. Undated uploads count as dated. DATED_AT reads documents.doc_time, but an undated upload stores now() with doc_time_source = 'upload_time'. Using an evidence date at all waits on A lease amendment chain still has four gaps after #679 and #680 #681 §1.
3. A slot conflict returns early. When the slot only records a conflict, the open rows are never compared, so C can stay open beside B.
4. Merge revert loses a fact. The slot re-cuts a correction the same merge made, and revert only restores surviving corrections.
5. Parallel extraction races. Nothing takes a lock and each rewrite commits separately: 39 of 40 concurrent trials left a duplicate live row.
6. The successor search for a start-less row doesn't require a start after the row's document date.

The redesign will:

• place a value between its predecessor and successor among all live rows, open and closed;
• run the reconcile in one transaction under an advisory lock on (kb, holder, predicate), with FOR UPDATE in close_superseded;
• make revert follow supersedes back to the rows from before the merge;
• exclude doc_time_source upload_time and none, if A lease amendment chain still has four gaps after #679 and #680 #681 §1 allows evidence dates at all.