migrate value-shapes: a lookup holding an embedded record object passes the scan — the exact case the ADR-0104 D1 gate exists to find

[baozhoutao]

Found while verifying the 17.0.0-rc.1 checklist on #3909 (F7, os migrate value-shapes). Verified on main @ 1ee48bc60 against a file-backed SQLite datastore.

os migrate value-shapes is the evidence half of the ADR-0104 D1 per-deployment gate: it is supposed to count every stored value strict mode would reject, so --apply can close the gate on real evidence. The scan's own header names the case it exists for:

a location stored as {latitude, longitude} or a lookup holding an expanded record object is APPLICATION data whose correct value only its author knows

That second case is not detected.

Repro

# a row the seeds do not own (see note below)
POST /api/v1/data/showcase_task {"title":"probe","project":"CB0-…","status":"backlog"} → id P

# plant the legacy embedded-reference form directly in storage
sqlite> update showcase_task set project='{"id":"CB0-…","name":"embedded"}' where id='P';

os migrate value-shapes --database-url file:legacy.db
→ Scanned 115 record(s) across 15 object(s) for reference / structured-JSON value shapes.
 ✓ No malformed values found.
 ✓ Scan clean. Re-run with --apply to record the flag and enforce.

sqlite> select project from showcase_task where id='P';
 {"id":"CB0-…","name":"embedded"} ← still there, still unreported

The row count confirms the record was visited (115 vs 114 before it existed).

Cause

valueShapeViolation parses the stored value with valueSchemaFor(def, 'stored'), and for every reference type that resolves to:

export const ReferenceIdValueSchema = lazySchema(() => z.string().min(1));

Any non-empty string passes — including the JSON text a legacy embedded reference is stored as in a TEXT column, which is exactly how such a value reaches a SQL deployment. The file half of the same family is strict and would catch it:

export const FileReferenceIdValueSchema = lazySchema(() =>
 z.string().regex(/^[A-Za-z0-9_-]{1,64}$/, 'Expected an opaque sys_file id'));

So lookup / master_detail references are the one class in the scan with no shape at all beyond "non-empty".

Impact

A deployment carrying exactly the values this gate exists to find runs the scan, is told it is clean, and closes the gate with --apply. The flag then attests a fact that was never checked. Because the scan deliberately imports the write-path predicate ("the scan must count exactly what strict mode would reject"), the write path is equally blind — so the value also survives future writes rather than being rejected once enforcement is on.

Expected

The stored form of a reference is an opaque id, so ReferenceIdValueSchema can be at least as strict as its file sibling (an id charset / length bound), which rejects JSON text, embedded objects, and stray whitespace while still accepting every real id the platform mints. If reference ids are deliberately free-form for external-key use, then the scan needs a separate heuristic for the embedded-object form its own docstring promises to find — otherwise the gate cannot deliver the evidence it is named for.

Note for whoever reproduces this

Any CLI command that boots the app re-runs seeds, and an upsert seed heals a planted value before the scan sees it — I lost a cycle to that. Plant into a record the seeds do not own.

Part of the #3909 rc.0/rc.1 verification.

[baozhoutao]

Tracked under the v17 verification tracker #3909.

[baozhoutao]

Rolled up in #4482 (all 17 defects from the v17 verification, grouped by severity with a suggested RC-exit triage).