Skip to content

fix(devx): give the driver-conformance census a dialect axis - #12134

Merged
os-warren merged 4 commits into
mainfrom
claude/issue-12014-conformance-census-dialect-axis
Aug 25, 2026
Merged

fix(devx): give the driver-conformance census a dialect axis#12134
os-warren merged 4 commits into
mainfrom
claude/issue-12014-conformance-census-dialect-axis

Conversation

@os-warren

@os-warrenos-warren commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Fixes#12014

The census scored driver × case-set and reported the same thing before and after a change whose entire content was "this conformance suite went from executing on one dialect to executing on three". ADR-0053 D-A3 declares the matrix with a third axis — driver {SQLite, Postgres at minimum} — and that axis was enforced only from inside a suite, by routing through live-dialect-matrix.testkit.ts and by OS_EXPECT_LIVE_DIALECT_MATRIX=1. Both are opt-in: they fire only for a file that already opted in. So a suite that hard-coded client: 'better-sqlite3' was invisible to every gate in the repo, counted here as a covered cell, with nothing anywhere saying it measured one dialect of three.

This adds the third axis to the census. DIALECTED asserts that a conformance suite says which dialects it runs on.

What moved

The covered-cell verdict is unchanged, and that is the point — no cell's CONSUMED answer moves, because nothing about coverage was wrong. What was missing is the second line:

check-driver-conformance: OK — 45 covered cell(s), 0 in the DEBT ledger, 0 exempt.
check-driver-conformance: dialect axis — 7 conformance suite(s) across 1 dialect-capable
driver(s): 5 run the matrix, 0 declare named cell(s), 2 in the DIALECT ledger. 8 of 9
dialect-scored cell(s) have a matrix-routed suite. 36 covered cell(s) belong to 4
single-backend driver(s), which have no dialect axis.

and the per-suite table above it:

dialect axis (ADR-0053 D-A3) — driver-sql speaks {sqlite, pg, mysql}
matrix sql-driver-aggregation-conformance.test.ts AGGREGATION
LEDGER sql-driver-comparand-type-conformance.test.ts FILTER_COMPARAND_TYPE
LEDGER sql-driver-icontains-and-retired-operators.test.ts FILTER_TEXT
matrix sql-driver-or-filter.test.ts FILTER_LOGIC
matrix sql-driver-pagination-conformance.test.ts PAGINATION, PAGINATION_UNORDERED, PAGINATION_ZERO_LIMIT
matrix sql-driver-temporal-conformance.test.ts TEMPORAL, TEMPORAL_TIME
matrix sql-driver-text-case-conformance.test.ts FILTER_TEXT

The classification counts

Re-measured on origin/main at e7480a8067, and they match the numbers the card was claimed with:

cardthis branch
files under driver-sql/src with a literal client: 'better-sqlite3'99105
files routing through live-dialect-matrix4044
files using dialectCell(77

The 105 are not 105 defects and this gate does not treat them as any. Most are legitimately SQLite-specific — sql-driver-11321-sqlite-audit-default-canonical.test.ts is about SQLite by name. The gap is that "deliberately single-dialect" and "accidentally single-dialect" were spelled identically.

The gate therefore scores a much smaller population: the suites that make a cell of this census covered — the ones whose coverage this script's own headline is asserting. That is 7 files, not 105:

  • 5 route through the dialect matrix.
  • 2 declare nothing, and are the seeded dialect ledger.
  • 1 further covering file (sql-driver.ts) is not a test file, so nothing executes and it carries no stance. It is named in the output rather than filtered away in silence.

Per-cell: 8 of 9 dialect-scored cells have a matrix-routed suite. The one that does not is FILTER_COMPARAND_TYPE_CASES, whose own CASE_SETS entry describes it as "the six accepted types compile everywhere" while it is measured on one dialect.

Why the two are ledgered rather than marked

Marking them dialectCell('sqlite') would be a mechanical, behaviour-preserving edit — the literal in both is byte-for-byte the sqlite cell's config(). It is deliberately not done here, because the marker would be recording a claim about intent that cannot be verified from the outside:

Both are a follow-up whose size this card's number sets: 2 suites, and the question for each is a behaviour decision (convert to the matrix, or declare the single cell), not a marking. No test file is edited by this PR.

Design decisions worth reviewing

Declared, not detected. The population is defined by the presence of a declaration, never by the absence of a client: literal. A detector keyed on the literal sees only the spellings it knows — move the config into a helper or behind a beforeAll and the literal is gone while the coverage is exactly as narrow. A suite passes by naming a stance symbol from its driver's testkit, so the gate cannot be respelled around.

Per file, not per cell. A cell can be covered by two suites. Scoring the cell would let an undeclared suite hide behind a matrix-routed sibling — which is the arrangement FILTER_TEXT_CASES is in on this tree right now.

Both axes still come off disk. Dialect-capable drivers are discovered by their DIALECT_CELLS export, and the cell ids are read from the testkit's array body — never a hardcoded package name or dialect list. driver-memory, driver-mongodb, driver-sqlite-wasm and driver-turso have no such testkit, so they are single-backend and are reported as a count rather than skipped in silence.

The comment mask is shared, not private. The stance is read from source text, so a mention must not count as a declaration — and on this tree the mentions are everywhere, because the files that were fixed describe the fix in prose. sql-driver-aggregation-conformance.test.ts (the #11456 conversion) quotes the import it now uses inside its head note; sql-driver.ts names FILTER_LOGIC_CASES and client: 'postgres' in comments only. Measured: 5 of this driver's 8 covering files change their client-literal answer between raw and masked text. The first cut of this branch carried its own stripComments; that is the exact anti-pattern scripts/js-comment-mask.mjs exists to delete, so it now routes through the shared module. The census output is byte-identical across that swap — a change of provenance, not of measurement.

The second ledger carries the same authority as the first. A second ledger is a second way to buy green, so DIALECTED's message carries the ⛔ MAINTAINER-ONLY label under the #8435 convention, with its own detector and its own three self-test assertions (the detector still reaches its subject; the real message carries the marker; an unmarked offer is rejected).

Ablation — the gate must be able to fail

Direction predicted in writing before the run: mutating the #11456 file so it no longer routes through the testkit turns the new gate red with exactly one DIALECTED error naming it, leaves the covered-cell count at 45, and leaves the baseline gate green.

Mutation: in sql-driver-aggregation-conformance.test.ts, the import specifier './live-dialect-matrix.testkit.js''./ablated-not-the-testkit.js'. This reproduces the pre-#11456 state at the level the census reads: still drives AGGREGATION_CASES, no longer routes through the matrix.

Proven on disk by anchored grep counts of the text actually changed, before any result was read — removed text present 0, injected text present 1.

treebaseline census (origin/main)this branch
cleanexit 0 — 45 covered cell(s)exit 0 — 45 covered cell(s) + dialect line
mutatedexit 0 — 45 covered cell(s)exit 1 — 1 problem, DIALECTED: … sql-driver-aggregation-conformance.test.ts

diff of the baseline census's two runs is empty: the pre-change gate's output is byte-identical on the clean and mutated trees. That is the defect of #12014, reproduced on this tree with a real mutation rather than argued.

Restore was verified, not trusted: trap … EXIT INT TERM, then git checkout HEAD -- "$TARGET" with an absolute path (never a repo-relative one after a cd, and never git checkout -- alone, which restores from the index the mutation step itself wrote). Then git hash-object of the working file returned e488edad7ae0994c9a5a3ed5ff7b890d1ef9087c, equal to git rev-parse HEAD:packages/drivers/driver-sql/src/sql-driver-aggregation-conformance.test.ts, and git status --porcelain was empty. The ablation was re-run after the comment-mask swap, since the detector changed.

The same red/green is permanent in --self-test, driven over a synthetic tree by the real dialectAudit rather than a re-implementation of it, along with the stance classifier in all three directions, the ledger reconciling in all three directions, and the non-vacuity guard that fails if the prose-only fixture ever stops being recognisable as a stance in raw text.

Verification

Gate union derived, not recallednode scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths; the script takes the changeset itself). All run at bc203848df, the final commit. Every exit code captured before any pipe; each verdict line below is the gate's own, not $?.

gateverdict
check:driver-conformanceOK — 45 covered cell(s), 0 in the DEBT ledger, 0 exempt. + the dialect line above; self-test OK
check:agent-test-spelling0 violations — 374 file(s) · 3808 bare -- token(s) …
check:cross-package-test-inputsOK: 16 package(s) read outside themselves, all declared …
check:entry-guard161 scripts/ file(s) — every entry guard goes through invoked-as.mjs …
check:parse-guard160 scripts/ file(s) — every TypeScript parse goes through ts-parse.mjs.
check:pnpm-filter-targets135/170 --filter occurrence(s) across 27 file(s) resolve …
check-ci-filter-parityOK: all 96 declared cross-package glob(s) (81 unique) are covered …
check-cross-package-test-inputsOK: 16 package(s) read outside themselves, all declared …
check:nul-bytesOK (scanned 6728 text file(s) … no raw ASCII control bytes).

Three families the derivation flagged as artifact rosters co-located with scripts/ — where a silent verdict "is not evidence in EITHER direction" — were run rather than reasoned about: check:i18n-stale-fill, check:pm-dispatch-gates (580 cases pass), check:partof-closing-keyword (28 cases pass). All green.

Because this PR now depends on the shared comment mask, its two owning gates were run too: js-comment-mask --self-test (35 cases pass) and check-comment-mask-corpus (5098 files, 0 disagree, 0 unparseable) — the sweep that diffs the mask against @typescript-eslint/parser across the tree.

npx eslint --no-inline-config --format json scripts/check-driver-conformance.mjs — 1 file, 0 errors, 0 warnings.

Changeset

None, skip-changeset label. The diff is one repo gate script; no published package changes and there is nothing to release. Graded against check-empty-changeset's own ledger, which is one-directional on exactly this case: an empty changeset satisfies the same gate the label does, produces no CHANGELOG entry (measured), and is uniquely a real input to changesets/action — the hasChangesets && !hasNonEmptyChangesets branch that silently stalled 17.0.0-rc.2 in #4898. It "buys nothing the label does not, and uniquely carries the risk".

The label was applied through the whole-set fallback rather than the additive endpoint: POST /issues/12134/labels returned HTTP 403 ("GitHub access is not enabled for this session"), the known per-session outage. So the current set was read (size/l), the union computed, and both written together. Read back after the write: see below.

Scope

Held to the declared surface: scripts/check-driver-conformance.mjs only. live-dialect-matrix.testkit.ts needed no widening — dialectCell() already exists and is the marker. No test file is edited, so the two held files (sql-driver-11224-update-stamp-precision.test.ts, sql-driver-11794-richtext-text-family.test.ts) and the three held sql-driver.ts bands are untouched.

Clause ② — no. A census script plus a convention marker; no public surface widens, no shipped accept/reject door moves. What a suite asserts about product behaviour is unchanged.


Generated by Claude Code

The census scored `driver x case-set` and reported 45 covered cells before and
after a change whose entire content was "this conformance suite went from
executing on one dialect to executing on three". ADR-0053 D-A3 declares the
matrix with a third axis -- `driver {SQLite, Postgres at minimum}` -- enforced
only from inside a suite, by opt-in routing through live-dialect-matrix.testkit
and by OS_EXPECT_LIVE_DIALECT_MATRIX=1. Both fire only for a file that already
opted in, so a suite hard-coding `client: 'better-sqlite3'` was invisible to
every gate in the repo while counting as a covered cell.
Measured cost, not hypothetical: sql-driver-aggregation-conformance.test.ts was
green on main and on every PR; on live PG 16.13 sum/avg/min/max over a boolean
column threw SQLSTATE 42883.
DIALECTED is the new invariant. It asserts that a conformance suite SAYS which
dialects it runs on -- not that the answer is good enough, which is a different
decision. The population is defined by the presence of a DECLARATION, never by
the absence of a client literal, so moving the config into a helper cannot
respell a suite out of the gate's reach.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6HFzyH98W1YaQXhJUJt6o
The UNDECLARED glyph is four characters wider than the others, so the run where
the table is most worth reading was the one where its columns broke. Found by
the ablation, which is the only run that produces that glyph.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6HFzyH98W1YaQXhJUJt6o
The first cut of this axis carried its own stripComments. That is the exact
anti-pattern scripts/js-comment-mask.mjs was written to delete: every
source-scanning gate used to answer "comment or code" privately, and the copies
drifted into two silent families -- a naive regex that opens a phantom comment
on a `/*` inside a string, and a string-aware scanner that opens a phantom
string on a quote inside a regex character class. The shared module is validated
against @typescript-eslint/parser over the whole tree by
check-comment-mask-corpus.mjs in CI; a private copy is validated by whatever its
author thought of.
stripComments is the documented projection for this caller: it feeds a scanner
and reports neither a line number nor a byte offset.
The census output is byte-identical across the swap, so this is a change of
provenance and not of measurement.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6HFzyH98W1YaQXhJUJt6o
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/lskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[finding] The driver-conformance census cannot see the dialect axis — a suite running on 1 of 3 dialects scores identically to one running on all 3

2 participants

@os-warren@claude