A mapping proposal has never been scored

[WaylandYang]

explore_mappings reads a source's schema, asks the model for up to twelve concepts and writes them as proposals. Nobody knows how good they are. There is no corpus, no truth, no score — while auto_type_resolution defaults to on because it was measured (39/41) and governance defaults to off because it was not. Every argument about confirming mappings automatically is currently an argument about a number nobody has.

Scoring an open generation

govern.mjs scores a binary verdict keyed by two names. A mapping proposal has no such key: the concept name is the model's own invention, so "Paid revenue" matches no truth entry written as "已支付 GMV".

Align on numbers instead. A truth entry is a business definition plus one gold SQL. Scoring runs the proposal — SELECT <expr> FROM <table>, or sql verbatim — against the same source and compares the result with the gold query's. Same number, same definition, whatever it is called. No per-proposal hand labelling, and the judgement is objective rather than a matter of reading two names.

Details that need deciding: float tolerance; a proposal matching several truth rows counts once and the rest are duplicates; dimensions carry no number, so they are judged by whether the column is in the truth's group-by set and whether the explorer filed it as a metric.

Four columns

• covered — of N truth definitions, how many some proposal computes
• right — of K proposals, how many run and match a truth definition
• wrong — run and match nothing, reported with the number produced and the nearest truth number
• broken — do not run at all (no such column, syntax)

wrong is the number that decides whether automatic confirmation can ever be default-on. A broken proposal is harmless: it fails loudly and a person sees it. A proposal that runs and is wrong carries the whole risk, because chat prints a number that looks entirely normal.

Corpora: TPC-H first, then TPC-DS

Both are TPC's decision-support benchmarks, and what makes them useful here is not their data generator but their published queries: TPC-H's 22 and TPC-DS's 99 each state a business question and answer it in SQL. sum(l_extendedprice * (1 - l_discount)) is not somebody's guess at what "revenue" means in this schema — it is the benchmark's own definition. That is exactly the shape a truth entry needs, and nobody can suspect the truth of having been written to match our implementation.

• TPC-H — 8 tables, a clean star with real foreign keys (l_orderkey → o_orderkey, o_custkey → c_custkey, c_nationkey → n_nationkey). Measures the basics and key recognition, and the customer → nation → region chain is the smallest case of one object whose fields live in three tables.
• TPC-DS — 24 tables, 99 queries, wider fact tables (store_sales at 23 columns) and several snowflaked dimensions. Closer to a real warehouse, and it exercises the proposal cap of Twelve proposals and eighty columns look the same #503 in a way TPC-H does not.

Carried on Postgres, not Trino

Trino ships both as built-in connectors, which would be the cheapest possible setup, but the image cannot be pulled here (Docker Hub is unreachable from this network — see the note in .env). Postgres turns out to be the better host anyway:

• Trino cannot answer The explorer cannot tell a key from a number #502. Its information_schema has no constraint tables, so a Trino-hosted corpus measures nothing about foreign keys. Postgres exposes them, and Postgres is the engine most bases actually use.
• Data volume is irrelevant. Exploration reads the schema and never the rows; the rows are used only at scoring time, where the gold query and the proposal run against the same tables. A few thousand rows from generate_series scores identically to SF1, and TPC's generator rules buy us nothing.
• Column comments become a variable we control. Real TPC-H has none, and comments are the model's main signal in a real base. Building the schema ourselves allows a commented and an uncommented variant of the same corpus, which measures directly how much the explorer leans on comments — impossible with the stock connector.

So: DDL and comments by hand from the benchmark specs, rows from generate_series + random(), truth derived from the published queries.

A third corpus — a wide table with deliberate traps (~60 columns; an integer store_id that sums cleanly and means nothing, an is_returned 0/1 whose sum is a real metric under a misleading name) — stays worth writing once these two have run, since neither benchmark flattens its joins the way a real reporting table does.

Shape

scripts/bench/mappings.mjs + truth/<corpus>.mappings.json, following the directory's rule: one fresh base per run. It matters more here than elsewhere — propose's ON CONFLICT … WHERE status = 'proposed' means a second run against the same base inherits the first run's rows.

Lands before the explorer changes, so those have a baseline.