feat(ci): caller inventory and drift guard (backend#1415)

[LukasWodka]

Closes part of tracebloc/backend#1415. Evidence: tracebloc/backend#1408.

What this adds

• repo-inventory.yml at the repo root — the single source of truth for which reusables each of the 20 active repos must call, plus its visibility and release-train membership.
• .github/workflows/caller-drift.yml — a report-only drift guard modelled on merge-settings-drift.yml, which is the fail-closed pattern the ticket names.
• scripts/caller-drift.py — the comparison, and scripts/tests/caller-drift-selftest.py — 42 offline assertions that the guard's failure paths actually fail. The audit job needs: the self-test.

The schema, and why absence fails

Every repo carries an entry for every reusable and every copy. Each entry is exactly one of:

state	meaning
required	the caller must be present (copies: content must match .github)
exempt: "<reason>"	the caller must be absent, and why is written down
divergent: "<reason>"	copies only: present, content knowingly differs

A missing key is a guard failure, not a default. So is an exempt whose reason is empty, null, whitespace, or absent; a bare exempt scalar with no reason; a repo in the org but not in the file; a repo in the file but not in the org; an unknown top-level key; a schema_version the guard was not written for; an audit_branch other than develop-first; an empty reusables list. Schema validation runs before any network call. There is no way to make the guard quiet by leaving something out — only by writing down why.

The inverse is also checked: a caller that exists where the inventory says exempt is reported as a stale exemption, and a divergent copy that has come back into line is reported so it can be promoted to required. Exemptions cannot quietly outlive their reason.

Matching is on uses: content, never on filename

The ticket's correction is right that a filename audit cannot identify a caller, though the specifics differ from what it says — see What the ticket got wrong below. Each workflow is parsed as YAML and its resolved uses: values are read. That also means code-quality.yml:60, a commented-out example uses: line, is not counted as a caller — a grep-based audit reports a code-quality caller in .github that does not exist.

Measured counts — develop-first, 2026-08-03

Taken via the API on develop where that branch exists, else the default branch. Twelve of twenty repos default to main/master, so a default-branch enumeration under-reports anything in flight.

110 caller references across 17 repos, all pinned @main. Three repos have no .github/workflows directory at all: devex-bootstrap, e2e-test-agent, rfcs.

repo	vis	train	adv	cq	cpb	fr-gate	fr-pass	closure	pii	set-pr	wip
.github	pub	–	✓	✗	✗	✗	✓	✓	✗	✓	✗
averaging-service	prv	✓	✓	✓	✓	✓	✓	✓	–	✓	✗
backend	prv	✓	✓	✓	✓	✓	✓	✓	–	✓	✗
claude-skills	prv	–	✗	✗	✓	✓	✗	✓	–	✓	✗
cli	pub	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗
client	pub	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗
client-runtime	prv	✓	✓	✓	✓	✓	✓	✓	–	✓	✗
data-ingestors	pub	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗
design-system	prv	✓	✓	✓	✓	✓	✓	✓	–	✓	✗
devex-bootstrap	prv	–	✗	✗	✗	✗	✗	✗	–	✗	✗
docs	pub	–	✗	✗	✓	✗	✓	✓	✓	✓	✗
e2e-test-agent	prv	–	✗	✗	✗	✗	✗	✗	–	✗	✗
frontend-app	prv	✓	✓	✓	✓	✓	✓	✓	–	✓	✗
model-zoo	pub	–	✓	✗	✓	✓	✓	✓	✓	✓	✗
release-train	prv	–	✓	✗	✗	✗	✗	✓	–	✓	✗
rfcs	prv	–	✗	✗	✗	✗	✗	✗	–	✗	✗
start-training	pub	–	✓	✗	✓	✓	✓	✓	✓	✓	✗
tracebloc-engine	prv	✓	✓	✓	✓	✓	✓	✓	–	✓	✗
tracebloc-py-package	prv	✓	✓	✓	✓	✓	✓	✓	–	✓	✗
tracebloc-website	prv	–	✓	✓	✓	✓	✓	✓	–	✓	✗

✓ present · ✗ absent (exemption written) · – structurally not applicable (private repo, so no public-pii-gate)

Per reusable: set-pr-status 17 · kanban-closure-router 17 · advance-deploy-env 15 · customer-priority-bump 15 · fr-pass-comment 15 · fr-gate 14 · code-quality 11 · public-pii-gate 6 · wip-limit-check0.

Filename conventions, measured:

reusable	caller filenames in use
code-quality	code-quality-caller.yml ×11
fr-gate	fr-gate-caller.yml ×14
fr-pass-comment	fr-pass-comment-caller.yml ×15
public-pii-gate	public-pii-gate-caller.yml ×6
customer-priority-bump	customer-priority-bump.yml ×15
advance-deploy-env	advance-deploy-env-caller.yml ×1 (.github), advance-deploy-env.yml ×14
set-pr-status	set-pr-status-caller.yml ×1 (.github), set-pr-status.yml ×16
kanban-closure-router	kanban-closure-caller.yml ×1 (.github), kanban-closure-routing.yml ×16 — matching neither pattern

Exemptions written — these are the findings

Today's state genuinely violates what the inventory should say, so current reality is encoded with an explicit reason on every gap. 220 entries in total: 140 required, 77 exempt, 3 divergent. The 80 non-required entries, grouped (19 + 20 + 31 + 10):

Structural — defensible, not gaps (19)

• public-pii-gate exempt on all 13 private repos: the gate scans PR text for PII that would be world-readable, and there is no public exposure to guard.
• fr-gate exempt on the 5 repos with no staging branch (.github, docs, devex-bootstrap, e2e-test-agent, rfcs) plus release-train, which drives the train and is exempt from riding it per RFC-BACKEND-1405 D1. No staging branch means no staging→prod hop to gate.

wip-limit-check exempt in all 20 repos (20) — it has zero callers anywhere in the org. Its own header says to delete the reusable once nothing references it. Requiring it anywhere would land this guard red on day one; wiring it up or deleting it is follow-up work.

Unremediated caller gaps (31) — each says UNREMEDIATED in so many words:

• code-quality ×9 — .github, claude-skills, devex-bootstrap, docs, e2e-test-agent, model-zoo, release-train, rfcs, start-training. Matches the ticket exactly.
• fr-pass-comment ×5 — claude-skills, devex-bootstrap, e2e-test-agent, release-train, rfcs. Cards there can only be advanced by hand.
• customer-priority-bump ×5 — .github, devex-bootstrap, e2e-test-agent, release-train, rfcs.
• advance-deploy-env ×5 — claude-skills, devex-bootstrap, docs, e2e-test-agent, rfcs. Merges there never move a card to On dev.
• kanban-closure-router ×3 and set-pr-status ×3 — devex-bootstrap, e2e-test-agent, rfcs.
• public-pii-gate on public .github ×1 — the only public repo without it. Sequenced behind backend#1409, since the reusable is currently unarmed org-wide (PII_DENYLIST unset).

Copy exemptions and divergences (10)

• add-to-kanban.yml and stale-backlog.yml exempt in devex-bootstrap, e2e-test-agent, rfcs (no workflows directory) — their issues and PRs never reach the board.
• stale-backlog.yml exempt in release-train — a deliberate omission: it is column-blind and would auto-close live pipeline work (backend#1408 P3).
• cli/add-to-kanban.ymldivergent — pins actions/add-to-project@v2.0.0 against v1.0.2 everywhere else.
• cli/stale-backlog.ymldivergent — pins actions/stale@v11 against v9 everywhere else.
• release-train/add-to-kanban.ymldivergent — adds a wanted explanatory header, and also pins add-to-project@v2.0.0.

Recording those three as divergent rather than required means the guard reports the drift it already found instead of hiding it behind a green run. Realigning the pins is follow-up work.

Fail-closed: every error path walked

Bugbot's recurring find in this epic is a failure path that reports success, or a precondition that fails open. For a drift guard that is the whole product, so every path was walked and is asserted by the self-test:

condition	result
inventory absent, unparseable, or not a mapping	exit 2
any schema violation above	exit 2, before any network call
org listing fails, or returns zero active repos	exit 2
release-train/repos.yml unreadable, or has an empty repos: list	exit 2 — an unreadable train list must not read as "nobody is on the train"
canonical copy missing from the checkout	exit 2 — otherwise every copy comparison passes
branch list 403 / rate-limited	repo UNREADABLE, run fails
repo has no branches; default branch not in the branch list	repo UNREADABLE
git tree unreadable, truncated, or has no tree array	repo UNREADABLE — a truncated tree must never read as "no workflows"
blob unreadable, wrong encoding, or tree entry with no sha	repo UNREADABLE
workflow is unparseable YAML	repo UNREADABLE — never "contains no caller"
every inventoried repo unreadable, or zero repos evaluated	exit 2
step outputs cannot be written	exit 2
the guard raises an unexpected exception	exit 2, not exit 1 — a crash is not a drift report
the audit step is skipped or crashes, so exit_code is empty	the final step's * branch fails the run

A partially-read repo is never evaluated: one unreadable file marks the whole repo unknown. The final workflow step runs if: always() and fails on anything other than a literal 0, including an absent value. It never posts an all-clear comment, only failures, and only on scheduled or manual runs.

Verified locally

• actionlint — clean.
• ruff check scripts/ — clean.
• Guard against today's real state: exit 0, all 20 repos read, no drift.
• scripts/tests/caller-drift-selftest.py — 42/42 pass, offline.
• 24 end-to-end fixtures against the live org: every schema violation exits 2; a false required, a stale exemption, a copy content drift, a visibility mismatch, a release-train mismatch, an untracked repo, a ghost repo, and a caller for an unlisted reusable each exit 1.

Deferred

The sync job that opens PRs for missing callers is deliberately not in this PR. Like merge-settings-drift.yml, this guard is report-only and never mutates a repo, so the token needs org-wide read rather than standing write over every repo. Follow-up work, in rough order:

1. The sync job itself — open a caller PR against any repo missing one it should have.
2. Decide wip-limit-check: wire it up, or delete the reusable.
3. Realign the three divergent copies to add-to-project@v1.0.2 / stale@v9.
4. Add public-pii-gate to .github, once backend#1409 arms the reusable.
5. Onboard devex-bootstrap, e2e-test-agent and rfcs, then flip their exemptions to required.
6. Normalise caller filenames — safe, since required checks reference job IDs (gate / gate), never filenames.

What the ticket got wrong

The measurement contradicts the ticket's own correction pass in five places. Conclusions all survive; the numbers do not.

1. "122 caller references across 16 repos" → 110 across 17. All @main, so that part holds.
2. "Four repos have no callers: devex-bootstrap, e2e-test-agent, release-train, rfcs" → three.release-train has three callers (advance-deploy-env.yml, kanban-closure-routing.yml, set-pr-status.yml). claude-skills — a repo the ticket does not mention at all — has four.
3. "fr-pass-comment ×7" → ×5.code-quality ×9, fr-gate ×6 and public-pii-gate on public .github are all confirmed exactly.
4. The naming premise is wrong in both the original and the correction. The correction lists code-quality as one of the inconsistently-named reusables; it is code-quality-caller.yml in all 11 repos. And for advance-deploy-env, set-pr-status and kanban-closure-router, the only repo using the -caller.yml form is .github itself — so the original ticket's ".github uses *-caller.yml, the others name the caller after the reusable" is closer to the truth than the correction that replaced it, except that fr-gate, fr-pass-comment, public-pii-gate and code-quality use -caller.yml everywhere. The operative conclusion — match on uses:, not filename — is unaffected and still correct.
5. wip-limit-check is listed as one of the reusables with callers under two filenames. It has no callers at all. That is a finding the ticket missed entirely: a reusable nothing references.

Two smaller corrections, both to backend#1408:

6. cli pins actions/stale@v11, not v10. The add-to-project@v2.0.0 drift is confirmed, and release-train has the same v2.0.0 drift — a second repo #1408 does not mention.
7. "model-zoo and start-training have fr-gate but no fr-pass caller" — both have fr-pass-comment-caller.yml today.

🤖 Generated with Claude Code