AIR consultation policy is contradictory: protocol says builder-decides, CLAUDE.md says one-consult-at-PR — pick one and align all surfaces

[amrmelsayed]

The contradiction (surfaced 2026-08-05/06 by two AIR runs)

• protocols/air/protocol.md (both trees): comparison table says AIR consultation is 'Optional (builder decides)'; PR-phase step 3 says 'Optionally run CMAP consultation if the builder judges the complexity warrants it.'
• CLAUDE.md/AGENTS.md (PIR section): describes PIR's consult footprint as matching 'BUGFIX/AIR's one-consult-at-PR pattern' — implying AIR consults at PR as standard.
• lessons-critical.md (hot tier, injected into every phase): 'Trust the protocol — never skip CMAP/consultation.'

Field consequence: air-1352's builder skipped consult per the protocol doc; the architect held the gate citing the lessons-tier rule; the builder accepted a 'misread' that wasn't one (corrected on PR #1355). air-1357 then did the same thing — correctly, per the doc. Enforcement is currently whichever surface the architect read last.

Evidence for the decision

air-1352's architect-requested consult on a 'trivial' build-script change caught two real stale arch.md build-wiring lines that both the builder's sweep and the architect's review missed — consistent with the hot-tier lesson's claim that consultation catches what solo review misses, even on small changes.

Options

A. Make AIR's consult mandatory-at-PR (align to BUGFIX's 'PR phase only'). Rationale: AIR is small features, BUGFIX small fixes — AIR being LIGHTER on review rigor than BUGFIX is inverted; the pass is one 2-way CMAP, cheap; it removes the ambiguity and matches CLAUDE.md's existing description and the lessons-tier rule. Cost: adds one consult (~1-3 min) to every AIR.

B. Keep builder-discretion and fix the other surfaces: correct CLAUDE.md/AGENTS.md's characterization, and qualify the lessons-critical line ('…where the protocol mandates it'). Cost: preserves a standing trap where the always-injected hot-tier lesson contradicts the protocol a builder is running, and leaves architects without a consistent gate-check expectation.

Recommendation: A. Whichever is chosen, align ALL FOUR surfaces in the same PR: protocols/air/protocol.md (both trees), the porch AIR protocol.json if it encodes consultation, CLAUDE.md+AGENTS.md, and — if B — lessons-critical.md (hot-cap displacement rules apply).

AIR-sized if option A (table row + step wording + porch config); BUGFIX-sized docs-only if option B.

[amrmelsayed]

Datapoint from the 2026-08-12/14 run, offered against my own recommendation on this issue.

I have argued for option A (consultation mandatory at the AIR PR gate), and the evidence for it kept accumulating: codex returned substantive REQUEST_CHANGES on a majority of the lanes in this window, catching real defects that the other two reviewers approved past — a masked blocked-gate label (#1428), an issue-closing Fixes on a partial PR (#1420), type-hiding mistaken for runtime restriction (#1411), an overview-nudge bug (#1410), and a false focus claim in shipped docs (#1073). During the ~6h codex outage the same window produced the reverse proof: PR #1442 shipped on two HIGH APPROVEs that missed an untested behavior change, caught only by architect source verification.

The counter-datapoint: PR #1452 (issue #1390) is a docs-only change — plugin rationale folded into a README section, pointer cleanup, plus a structure test. Consultation was skipped on that lane, and the skip is defensible: a three-model review of prose has a poor cost/benefit ratio, and mandating it would mostly train agents to treat the requirement as ceremony to satisfy rather than a check to use — which is how a mandatory gate quietly becomes worthless.

What this suggests for the ruling: the useful axis is not "AIR always consults" vs "AIR never consults" but what kind of change is in the diff. A rule of the shape "mandatory when the PR touches executable code or contracts; optional (owner's call, recorded on the PR) when it is docs/comments/assets only" would capture every real catch listed above while leaving prose-only lanes cheap.

Where that rule is hard, by the same example: PR #1452 adds a test file alongside its docs, so it sits exactly on the boundary the carve-out would create. Docs changes that also touch a manifest, a config value, or a test harness are executable in effect and should not inherit the docs exemption. Whoever writes the rule has to answer the mixed case explicitly, or every lane owner will answer it privately — which is the failure this issue already documents.

A compensating check worth writing into the rule. The real failure mode of documentation is factual drift from the system it describes, which a CMAP pass is not especially good at catching; a claim-by-claim source verification is. On this lane that check was performed and recorded on the PR, and it did more than a consult would have: it turned up a stale comment in an unrelated file (apps/vscode/src/command-relay.ts:78 still described a live workspace guard as an inert no-op). If docs lanes are exempted from consultation, the exemption should carry that obligation rather than nothing.

Whatever is decided, the current state — the protocol text and the operating practice disagreeing, with each lane owner resolving it privately — is the part worth ending. The consistency matters more than which side of the line docs land on.

(Edited: my original version attributed the consultation skip to the lane's architect as a scoping decision. That was wrong — the builder skipped it and the architect ratified the decision explicitly at review time, with reasoning recorded on the PR. Correcting it here because a decision record should say who actually decided what. The mixed-PR boundary and compensating-check paragraphs were added in the same edit.)

[amrmelsayed]

New evidence, and it is a different argument for consultation than the one this issue has been weighing.

The case so far has been about catching defects the author missed. PR #1468 (pir-1465, 2026-08-15) shows consultation catching something no author could have missed, because it was not wrong when written: codex and claude both flagged documentation that had been true when approved and was made false by the lane under review. One of the flagged passages was text I had personally approved in #1463 days earlier. Neither architect on that lane spotted it; both had already accepted the text once, and re-reading your own prior approval is exactly the thing a human reviewer is worst at.

The lane owner had triaged all three findings as "documentation/coverage rather than the core change". On verification, one of them would have shipped actively wrong guidance — so the "these are nits" reading was the dangerous one.

Why this bears on the ruling: an approval is a snapshot, our artifacts carry no expiry, and nothing in the system notices when the world moves under one. Three instances of that drift landed in a single week, each caught by a different accident — one by an architect's own re-read, one by chasing a rename, this one by consult. There is no reliable detector, and the partial mechanical one available (flag paths named in prior approvals that no longer exist) would have caught only the rename case.

That makes routine, cheap re-reading the only mechanism that has ever worked, and a consultation pass at the PR gate is the only place we currently get one systematically. It is an argument for mandating consultation that survives even if the code in a given PR is flawless — the review is not only of the diff, it is of what the diff has quietly invalidated.

It does not change my recommendation on the axis (mandatory when the diff touches executable code or contracts; owner's call, recorded, for docs/assets only). It does argue the reason for the mandate is broader than defect-catching, and that a PR whose diff is small or "just docs" is not therefore a safe place to skip it.