Manifest-driven validate(): the schema block is the spec, shared by the builders and a PR check

.github/workflows/consumed-file-check.yml

@@ -1,6 +1,8 @@
 # Go-live guardrail (PLAN Phase 5): a PR must not break a file a live lecture
-# consumes. The narrowest possible check — full PR validation (manifest schema,
-# dtypes, invariant tests) comes later in Phase 5 and will subsume this job.
+# consumes. This job is the byte-integrity (sha256) and catalog-freshness gate;
+# its sibling validate-datasets.yml (2026-09-07, #119) is the schema gate —
+# manifest conformance and per-dataset invariants. Together they are the "PR
+# validation" Phase 5 promised; neither subsumes the other.
 
 name: consumed-file-check
 

.github/workflows/validate-datasets.yml

@@ -0,0 +1,42 @@
+# PR validation (PLAN Phase 5; QuantEcon/data-lectures#119): the manifest's
+# schema block is an executable contract, and this is where it executes on
+# every PR. Two passes, both in scripts/validate_datasets.py:
+#
+#   conformance  every manifest obeys manifest-schema.yml's rules -- canonical
+#                dtype names (#122), name-xor-pattern with one capture group
+#                (#120), exact known_nulls and a `nulls:` placement block on
+#                every dynamic snapshot (#121), known_nulls_total only inside
+#                header-less sheet reads
+#   bytes        every CSV's committed bytes satisfy its own schema block,
+#                through the same builders/_validate.py the dynamic builders
+#                run -- so a manifest that drifts from its file fails here
+#                before it can fail a refresh
+#
+# Non-CSV formats get the conformance pass only, and the log says so.
+# consumed-file-check.yml stays beside this as the byte-integrity (sha256) and
+# catalog-freshness gate; this job is the schema half PLAN Phase 5 promised.
+#
+# pandas is pinned to requirements.txt's version (2.3.3), deliberately one
+# major behind the lectures' anaconda=2026.07 (pandas 3): the validator
+# compares dtypes by family precisely so it is green on both, and running the
+# older one here is the standing check that it stays so.
+
+name: validate-datasets
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          lfs: false          # lectures/ is plain git; a pointer here should fail (see consumed-file-check.yml)
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install pandas==2.3.3 PyYAML==6.0.3
+      - run: python scripts/validate_datasets.py

AGENTS.md

@@ -122,7 +122,7 @@ Where one builder produces a **set** of files, name it for the set and let each
 
 **Where a builder reads its input from.** The normal case is the third-party upstream, fetched at run time: eight of the nine `committed` builders here do that, and it is the fetch stage of the contract below. A builder reads from `sources/` **only when the input cannot be re-fetched** — the upstream is gone, unlocatable, or was inherited with no recoverable source. `sources/` is that exception layer, not a general input tree, and it is emphatically not "the big-file directory": the defining property is un-refetchability, not size. What it must never be is a network read from another QuantEcon repo — that is how a retired repo becomes load-bearing again.
 
-Builders follow four stages — **fetch → pre-process → validate → write** — and only write on validation pass (expected columns/dtypes, row-count floor, recency of date range, no all-NaN columns, and a **bounded** overlap window against the previous vintage — a tracking snapshot is revised by its source, so the test is a tolerance plus a printed summary, never equality). Lectures always read the last-good snapshot: an upstream outage may fail a refresh, it must never break a lecture build.
+Builders follow four stages — **fetch → pre-process → validate → write** — and only write on validation pass. **The validate stage is shared**: `builders/_validate.py` reads the manifest's `schema` block as its spec (columns and `pattern` runs, dtype families, exact `known_nulls`, the `nulls:` placement rule, `row_count_floor`, `date_range`) and measures the overlap window against the previous vintage; a builder calls `validate(frame.reset_index(), manifest, previous)` and layers on only what the schema cannot say — value bands, a grid check, the per-series revision **bound** (a tracking snapshot is revised by its source, so the test is a tolerance plus a printed summary, never equality). The same function runs over every committed CSV on every PR (`scripts/validate_datasets.py`, `validate-datasets.yml`), so a manifest that drifts from its bytes fails the PR, not the next refresh. Lectures always read the last-good snapshot: an upstream outage may fail a refresh, it must never break a lecture build.
 
 **One builder per source, composite files where a lecture reads series together, shared fetch libraries** (decided 2026-09-01, #26): `builders/_fred.py` is the `Fred` class for FRED reads — no `pandas_datareader` in a builder — and a lecture's FRED data is one file with the lecture's own variable names as columns, not six. A builder that writes a set validates every file before writing any.
 

PLAN.md

@@ -296,7 +296,7 @@ Only one file genuinely forces LFS, and it is not a dataset:
 Full automation:
 
 - [x] Audit dashboard workflow ([#20](https://github.com/QuantEcon/data-lectures/issues/20), added 2026-07-17): `.github/workflows/audit-dashboard.yml` rebuilds the full-universe data audit + migration tracker from the 8 lecture repos' `main` (push to main / weekly / dispatch) and deploys it with the published tree to Pages. Strict mode fails the build on an unannotated data reference or a `migration.yml` status the scan contradicts
-- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — have their own issues under the [#14](https://github.com/QuantEcon/data-lectures/issues/14) tracker: [#120](https://github.com/QuantEcon/data-lectures/issues/120) column patterns, [#121](https://github.com/QuantEcon/data-lectures/issues/121) `known_nulls`, [#122](https://github.com/QuantEcon/data-lectures/issues/122) dtype vocabulary; the shared manifest-driven `validate()` and this workflow are [#119](https://github.com/QuantEcon/data-lectures/issues/119)
+- [x] PR validation — manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — have their own issues under the [#14](https://github.com/QuantEcon/data-lectures/issues/14) tracker: [#120](https://github.com/QuantEcon/data-lectures/issues/120) column patterns, [#121](https://github.com/QuantEcon/data-lectures/issues/121) `known_nulls`, [#122](https://github.com/QuantEcon/data-lectures/issues/122) dtype vocabulary; the shared manifest-driven `validate()` and this workflow are [#119](https://github.com/QuantEcon/data-lectures/issues/119) **Landed 2026-09-07 (#119)**: `builders/_validate.py` reads the sidecar `schema` block as its spec and is shared by the dynamic builders and by `.github/workflows/validate-datasets.yml`, which runs `scripts/validate_datasets.py` on every PR — conformance for all 44 manifests, byte validation for every CSV; non-CSV formats are conformance-only until their range reads are built.
 - [x] Retrofit `builders/business_cycle.py` to the four-stage builder contract — **done 2026-09-01**, with the two provenance dumps moved out of the published tree to `provenance/` ([#13](https://github.com/QuantEcon/data-lectures/issues/13)). Its `validate()` is the first to face a *revised* upstream: it bounds the overlap window (5 pp) and prints the revision summary rather than asserting equality, which is the review surface the refresh-as-PR workflow below will use. It previously had fetch/transform/write but no validate stage. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14)
 - [x] Scheduled refresh workflow for dynamic datasets — **landed 2026-09-01** as `.github/workflows/refresh-snapshots.yml`, manifest-driven rather than cron-per-class: a weekly run asks `scripts/snapshots.py due` which `dynamic-snapshot` datasets have their cadence elapsed (or are `diverged`, or were never refreshed), runs each builder in place, stamps the manifest (`retrieved`, `sha256`, `integrity.upstream: verified`, `date_range.end`), regenerates the catalog, and opens a PR on `refresh/<stem>` whose body is the builder's overlap summary. Nothing auto-merges; the first consumer is `business_cycle_data.csv`, not UNRATE — the pilot's order inverted once the World Bank file turned out to be the one already here
 - [x] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — **landed 2026-09-01** as the `canary` job of the same workflow: every dynamic snapshot's builder runs with `--out-dir`, and a failure opens or updates one `upstream-break` issue classified by exit code (2 = the data broke the contract, a human; anything else = the fetch, a retry). Covers the live APIs only as their snapshot twins land here — the 23 live-API lectures without a twin are still guarded by nothing but their own CI

builders/README.md

@@ -30,7 +30,16 @@ template is [`_template.py`](_template.py) (not a builder — the underscore
 keeps it out of any manifest). Shared fetch code lives beside it under the
 same convention: [`_fred.py`](_fred.py) is the `Fred` class every FRED
 builder should use (`fred.series('UNRATE')`, `fred.frame([...])`), so a fetch
-stage is a line and `validate()` is the only thing worth reading. One
+stage is a line and `validate()` is the only thing worth reading — and most of
+*that* is shared too: [`_validate.py`](_validate.py) reads the manifest's
+`schema` block as the spec (columns and `pattern` runs, dtype families, exact
+`known_nulls`, the `nulls:` placement rule, `row_count_floor`, `date_range`)
+and measures the overlap window against the previous vintage
+([#119](https://github.com/QuantEcon/data-lectures/issues/119)). A builder
+calls `validate(frame.reset_index(), manifest, previous)` and adds only what
+a schema cannot say — value bands, a grid, recency, the revision *bound*. The
+same function runs over every committed CSV on every PR
+(`scripts/validate_datasets.py`, `validate-datasets.yml`). One
 builder per **source** for a lecture's data, writing a composite file where
 the lecture reads the series together (decided 2026-09-01 on #26).
 

builders/_template.py

@@ -32,7 +32,7 @@
               (an end year, an observed range, a row count) -- those live in
               the fields scripts/snapshots.py stamps, and only there
 
-Requires pandas plus whatever the source needs (add it to requirements.txt).
+Requires pandas and PyYAML plus whatever the source needs (add it to requirements.txt).
 """
 import argparse
 import datetime as dt
@@ -41,6 +41,9 @@
 import sys
 
 import pandas as pd
+import yaml
+
+from _validate import ValidationError, validate as validate_schema
 
 CURRENT_FILE_DIR = os.path.dirname(os.path.abspath(__file__))
 REPO_ROOT = os.path.dirname(CURRENT_FILE_DIR)
@@ -52,10 +55,6 @@
 MAX_STALENESS = None             # newest observation must be at least this recent
 
 
-class ValidationError(Exception):
-    """The fetched data broke the published contract -- exit code 2."""
-
-
 def _check(condition, message):
     if not condition:
         raise ValidationError(message)
@@ -73,22 +72,32 @@ def pre_process(raw):
 
 
 def validate(frame, previous=None):
-    """Assert the contract the manifest's schema block promises; return the
-    run summary. Every failure is a ValidationError with a message a human
-    can act on from the canary issue."""
-    _check(len(frame) > 0, 'empty frame')
-    # columns / dtypes / known_nulls / units / recency ...
+    """Two layers (QuantEcon/data-lectures#119). The manifest's schema block
+    is the spec -- columns and pattern runs, dtype families, exact known_nulls,
+    the `nulls:` placement rule, row_count_floor, date_range, and the overlap
+    window MEASURED against `previous` -- enforced by builders/_validate.py.
+    Add here only what a schema cannot say: value bands, a grid, recency, and
+    the revision BOUND. Every failure is a ValidationError with a message a
+    human can act on from the canary issue."""
+    with open(os.path.join(PUBLISHED_DIR, OUT_FILE + '.yml')) as f:
+        manifest = yaml.safe_load(f)
+    # pass the RAW shape: the period/label column as a column, not the index
+    raw = frame.reset_index() if frame.index.name else frame
+    prev_raw = previous.reset_index() if previous is not None and previous.index.name else previous
+    shared = validate_schema(raw, manifest, prev_raw)
+    # builder-specific: bands / grid / recency ...
+    raise NotImplementedError
     summary = {
         'dataset': OUT_FILE,
         'builder': os.path.relpath(os.path.abspath(__file__), REPO_ROOT),
-        'rows': int(frame.shape[0]),
-        'columns': int(frame.shape[1]),
-        'date_range': {'start': None, 'end': None},
-        'overlap': None,
+        'rows': shared['rows'],
+        'columns': shared['columns'],
+        'date_range': shared['date_range'],
+        'overlap': shared['overlap'],
     }
     if previous is not None:
-        # compare the shared window; report and BOUND revisions, never assert equality
-        raise NotImplementedError
+        _check(shared['overlap']['max_abs_change'] <= MAX_REVISION,
+               f'revision {shared["overlap"]["max_abs_change"]} exceeds {MAX_REVISION}')
     return summary