Phase 5: the weekly canary and refresh-as-PR for dynamic snapshots

[mmcky]

The two Phase 5 automation boxes that the P4 groundwork (#109) made possible, as one manifest-driven workflow: .github/workflows/refresh-snapshots.yml. Written to the policy recorded in AGENTS.md ("Refresh, break, or schema change — who gets told").

What runs, weekly (Monday 06:17 UTC) and on dispatch

Job	Does	On failure
plan	scripts/snapshots.py list and due → two matrices	—
canary	every class: dynamic-snapshot builder with --out-dir: fetch + validate, no commit	notify opens or updates one upstream-break issue, classified by exit code — 2 = ValidationError (the data broke the contract; a human) vs anything else = the fetch (a retry) — with the builder's log tail. Same shape as the audit's drift issue. No lecture is affected either way: consumers read the last-good snapshot
refresh	only datasets that are due — cadence elapsed since retrieved, or integrity.upstream.status: diverged, or never refreshed — and only if the canary passed: builder in place → manifest stamped → CATALOG.md regenerated → check_consumed_files → a PR on refresh/<stem> whose body is the builder's overlap summary. A later run updates the same branch and PR	—

scripts/snapshots.py is the manifest side: list, due, stamp, pr-body. stamp edits the manifest text rather than a parsed-and-re-dumped copy (PyYAML would discard every comment the sidecars carry their reasoning in), replaces each stamped key line plus its continuation with a single-line value, drops the diverged delta block, and then re-parses the file and refuses to write unless every stamped field reads back as intended.

Builder contract additions, in builders/business_cycle.py and the copy-able builders/_template.py (#14's "when Phase 5 is actioned", step 1): --summary-json, a ValidationError with exit code 2, validate() returning the run summary. The manifest is made stampable — its title no longer embeds the end year, and every stamped field is a single-line value with its reasoning in comments above it, not beside.

Docs. PLAN Phase 5: scheduled refresh and canary ticked; consumer fan-out left open with the on_refresh policy noted (no snapshot has a consumer yet). AGENTS.md: the refresh contract under "Builders" and snapshots.py in the repo map. builders/README.md likewise. README.md: the layout table still said builders lived in scripts/; it now lists builders/, sources/, provenance/ and scripts/ correctly, plus a "Dynamic snapshots" section.

Token. The refresh PR is opened with QUANTECON_SERVICES_PAT, which is what lets it trigger the required consumed-files check (GitHub does not run pull_request workflows for PRs opened by the workflow token). Access for this repo was added on 2026-09-01. The workflow falls back to the workflow token if the secret is ever absent, and the PR body says so with the close-and-reopen workaround.

Tested locally end to end: due → builder --summary-json → stamp → the manifest read back verified with the new hash and date_range.end: 2025, check_consumed_files clean, due flipped to "skip", pr-body rendered; the stamped state was then reverted, so this PR changes no published bytes. check_consumed_files 0 errors on the branch; strict audit problem set unchanged.

What happens after merge. business_cycle_data.csv is diverged, so it is due: the first run will open the real first refresh PR — 236 of 320 overlap cells revised, YR2024–YR2025 added — for review through its overlap summary. Suggest a workflow_dispatch straight after merging rather than waiting for Monday, so the whole loop runs with someone watching.

See #14.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Implements PLAN Phase 5 automation for class: dynamic-snapshot datasets by adding a manifest-driven snapshot “planner”, a weekly canary + refresh-as-PR GitHub Actions workflow, and updating builder/docs to support the refresh contract.

Changes:

• Add scripts/snapshots.py to list dynamic snapshots, compute which are due, stamp manifests from builder summaries, and generate refresh PR bodies.
• Add .github/workflows/refresh-snapshots.yml to run weekly canaries and open/update refresh/<stem> PRs for due snapshots.
• Update builders/business_cycle.py (and add builders/_template.py) to emit --summary-json and use exit code 2 for ValidationError, plus accompanying documentation updates.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/snapshots.py	New CLI for manifest-driven due-logic, stamping, and PR-body generation for refresh automation.
.github/workflows/refresh-snapshots.yml	New scheduled/dispatch workflow running canary + refresh-as-PR for due dynamic snapshots.
builders/business_cycle.py	Extends builder to produce --summary-json and provide exit-code semantics for canary classification.
builders/_template.py	Adds a copyable builder template documenting the refresh contract.
lectures/business_cycle_data.csv.yml	Makes the manifest stamp-friendly and updates title/description wording for dynamic end-year.
CATALOG.md	Regenerated catalog reflecting the updated dataset title text.
README.md	Corrects repo layout description and documents dynamic snapshot behavior.
builders/README.md	Documents dynamic snapshot builder contract and points to the template.
PLAN.md	Marks Phase 5 workflow + canary as landed and updates the Phase 5 plan text accordingly.
AGENTS.md	Updates builder guidance and repo map to include snapshot plumbing and contract details.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.