Schema decision: is known_nulls an exact count or a ceiling?

[mmcky]

schema.known_nulls maps a column to a null count, and #14's template treats it as exact: n != known.get(col, 0) fails. Two dynamic snapshots now carry counts that a refresh will change:

• business_cycle_data.csv.yml: {YR1960: 9} — stable by construction, growth is undefined in the first year for every economy.
• the FRED composite: {UNRATE: 349, UMCSENT: 616, CPILFESL: 457, M0892AUSM156SNBR: 1132} — leading nulls before each series starts, on a monthly panel whose length grows with every refresh, so at least some of these counts move as soon as the panel is extended.

Frozen extracts want exact counts (a changed null is a changed file). Tracking snapshots want a shape constraint, not a number.

Options.

1. Exact everywhere; the refresh stamps the new counts alongside retrieved and date_range.end, so the manifest diff shows them. Simple, but it turns a data invariant into a field the pipeline rewrites.
2. Ceiling everywhere: n <= known. Never fails on a refresh that fills in history, but it silently accepts a column going empty below the ceiling.
3. Exact by default, with an explicit ceiling form for tracking snapshots, e.g. {UNRATE: {max: 400}} or a {col: "<=400"} string. Two forms to document and validate.
4. A positional rule instead of a count: nulls allowed only before a series' first observation or in the newest k rows, never inside (what business_cycle's validate() already implements by hand), with the count kept as documentation.

Decide, then update manifest-schema.yml's known_nulls comment and the template's check in step with the manifest-driven validate().

Sub-issue of #14.

[mmcky]

Decision, 2026-09-07: the integer stays exact; dynamic snapshots declare null placement, not a count

Option 4 from the body, chosen after measuring what the manifests declare and what the two dynamic builders already assert.

What is in the tree today. 12 of 41 manifests declare non-empty known_nulls, all as plain integers; ten are static files where exact is right. Two manifests (unemployment_rate_annual.csv.yml, private_credit_to_gdp.csv.yml) use a third form, known_nulls_total: N, described in their own comments as a ceiling — and that key is declared nowhere: not in manifest-schema.yml, not in any script. So three shapes are in the wild and one is documented.

Which counts actually move on a refresh. The FRED composite's nulls split by position (first vintage, 1919-01 to 2026-07):

column	total	leading	inner	trailing
UNRATE	349	348	1	0
UMCSENT	616	406	210	0
CPILFESL	457	456	1	0
M0892AUSM156SNBR	1132	123	0	1009

Only M0892's count grows, by one per month, because the series ended in 1942-06 and every new row adds a trailing null; the other three are stable unless the source backfills. The inner nulls are the 2025-10 shutdown hole and UMCSENT's sparse pre-1978 years. business_cycle_data's {YR1960: 9} is stable by construction; the two World Bank siblings have leading nulls for GBR and FRA and one trailing null for an unpublished newest year. Neither dynamic builder asserts a count — both assert placement (before first observation, in known inner holes, after last observation), and the manifest comments say the counts are "recorded for the reader, not asserted".

The rules.

• A plain integer under known_nulls is exact: n != known fails. Required for any nulled column in a verbatim or constructed file.
• A dynamic-snapshot declares placement per column with a small closed vocabulary — leading, trailing, and inner: [...] listing the known holes (dates for a time index, column names for a wide file). The validator checks that every null sits where declared and nowhere else. A dynamic file may still write an integer for a column whose count is stable (YR1960: 9) to get the stronger check.
• The default is by class, which every manifest already carries — no new per-file judgement field.
• known_nulls_total is retired: a file-level total cannot say where the nulls are. Its two users move to leading on the affected columns and trailing on the newest year.
• No {max: N} ceiling form. A ceiling silently accepts a column emptying out below the bound, which is the failure the check exists to catch.

Edits this implies. Placement blocks in the four dynamic manifests, the known_nulls comment and a nulls placement entry in manifest-schema.yml, and the template's check replaced by the shared one from #119. Closes with the PR that records the vocabulary in manifest-schema.yml.

[mmcky]

Correction to the decision above, found while writing it into the tree: known_nulls_total is used by four manifests, not two. Besides the two World Bank siblings (where it was a ceiling and is now replaced by a nulls: placement block), it appears nine times inside the sheets: entries of assignat.xlsx.yml and dette.xlsx.yml. Those are positional reads with header: null, so the range has no column names to count against, and a file-level exact total is the only form a count can take. There it is not a ceiling and not drift — it is an exact count for a static file, which is exactly what the integer rule wants.

So the rule as recorded in manifest-schema.yml: known_nulls_total: N is legal only inside a header: null sheet entry, as an exact total; it is retired for named columns and never a ceiling. The rest of the decision stands unchanged.