"Unknown review type" blames porch config when the real fault is an unbuilt skeleton/ tier

[pseudoseed]

Found by builder-bugfix-196 on 2026-08-29 while diagnosing 120 test failures across 23 files
in a worktree whose only sin was not having been built yet.

The misleading part first

The failure reads:

Unknown review type "pr" in porch.consultation.modelsByType.
Review types declared by the protocols available here: "impl".

That message names porch.consultation.modelsByType — a config key — as the culprit. It is
not the culprit. The config is correct and untouched. Three separate hypotheses were chased on
the strength of that sentence (an edited .codev/config.json, a leaked builder environment, and
the globally installed package skeleton) before anyone questioned the message itself.

This is worse than a check that cannot tell. A check that reports "could not determine" sends
the reader looking. This one sends the reader confidently to a file that is correct, where they
stay.

The actual cause

packages/codev/skeleton/ is a build artifact, not tracked in git:

"copy-skeleton": "rm -rf skeleton && cp -r ../../codev-skeleton skeleton"

getSkeletonDir() resolves path.resolve(__dirname, '../../skeleton'), and that is tier 4 of
the four-tier resolver (.codev/ → codev/ → runtime cache → installed package skeleton).
protocolDirs() ends with getSkeletonDir()/protocols; listProtocolNames unions all four
tiers; listReviewTypes resolves per name.

In a worktree where only dependencies were built (pnpm --filter '@cluesmith/codev^...' build
excludes codev itself), packages/codev/skeleton does not exist, tier 4 is empty, and
knownTypes collapses to whatever the individual test fixture declares.

Verified side by side:

ls packages/codev/skeleton -> No such file or directory (unbuilt worktree)
ls .builders/spir-146/packages/codev/skeleton -> exists (built worktree)
ls packages/codev/skeleton -> exists (main checkout)

That is also the scenario-4-vs-6 split: scenario 6's beforeEach writes
writeProtocol('spir', ['impl']) and its body writes modelsByType: { pr: [...] }, so pr can
only be known via the skeleton tier. Scenario 4 declares all four itself and passes. Scenario 6
has a hidden dependency on a build artifact, and nothing declares it.

Why nobody has hit this

CI builds before it tests. A developer who runs the full build has it. Only a worktree built
partially — dependencies but not codev itself — lands in this state, and that is exactly what a
fresh builder worktree looks like before its first full build.

Ask

1. Fix the message. When a review type is unknown and the skeleton tier is absent, say the
   tier is absent. Naming the config as the cause when the resolver is empty is the defect; the
   missing artifact is only the trigger.
2. Assert the dependency. Tests that depend on the skeleton tier should fail immediately and
   by name when getSkeletonDir() does not exist — one existsSync in the affected suite's
   setup — rather than 23 files failing with a message that points elsewhere.
3. Consider whether the resolver itself should distinguish "tier 4 is empty because nothing is
   installed" from "tier 4 has no protocol by that name". Those are different facts and currently
   produce the same answer.

Point 1 is the ask. Points 2 and 3 are how it stops recurring.

Related

This is the third door found today into "the suite reads state from outside the tree":
the symlinked .codev/config.json, the leaked CODEV_* builder environment (#189), and now an
untracked build artifact. See #199 — a check must report whether it was in a position to observe.

[pseudoseed]

Adding the full message set from bugfix-196's run. It is worse than three: five distinct messages
for one root cause, and a fourth build artifact is involved — porch-driver/dist, a different
package's output.

One root cause: the tree was never built (pnpm build produces packages/codev/skeleton/,
packages/codev/dist/, and the sibling packages' dist/). 20 test files, 80 failures, five stories:

#	message	verdict
1	Cannot find module '…/packages/codev/dist/terminal/shellper-main.js'	honest — names the file
2	Cannot find module '../../../../../porch-driver/dist/thread.js'	honest — names the file
3	Roles directory not found in .codev/roles/, codev/roles/, or embedded skeleton	misleading
4	Skeleton directory not found. Package may be corrupted.	wrong
5	Unknown review type "pr" in porch.consultation.modelsByType. Review types declared by the protocols available here: "impl".	wrong subsystem

The two that name a missing file are fine — a reader follows them straight to the cause.

#3 lists the three places it looked and never mentions that the fourth, the embedded skeleton, is
generated by the build. A reader checks all three named paths, finds nothing wrong, and stops.

#4 accuses the installed package of corruption. That sends a reader to reinstall the package,
which cannot help, and costs them a download before they learn nothing.

#5 is the expensive one, and it is the pattern this issue leads with. It names
porch.consultation.modelsByType as the culprit. I followed it to .codev/config.json, found the
file correct, and spent the next stretch investigating a config that was never wrong — including
building a hypothesis about a symlinked workspace config that turned out to be unrelated. The
mechanism: listReviewTypes() unions protocol names across all four resolver tiers but resolves each
protocol.json by precedence, so an empty tier-4 (unbuilt skeleton/) silently shrinks the known
review-type set to whatever the test fixture itself declares. The config key it names is a victim
of the empty tier, not the cause.

That is the distinction worth holding onto: #1 and #2 are checks that could not tell. #3 and #4 are
checks that told a reader to look somewhere harmless. #5 is a check that confidently told a reader
the wrong thing, and it is strictly the most expensive of the three kinds because the file it accuses
is right there and looks fine.

Suggested shape, since all five share one precondition: whatever detects an unbuilt tree should say
so once, by name, before any of these five can fire — packages/codev/skeleton/ is missing; run pnpm build. Any message that can be caused by an unbuilt tree and does not say "unbuilt tree" is
guessing.

Evidence: full-suite run in .builders/bugfix-196 at 12:07 on 2026-08-29 — 20 files / 80 tests,
every one traceable to a missing build artifact except one unrelated regression of my own.

---

A fourth point, because the fourth artifact is a different defect from the other three.

skeleton/, dist/ and the roles directory are all produced by pnpm build in
packages/codev. packages/porch-driver/dist is not — it is a sibling package, and
building codev does not build it. So a developer who runs the documented build still ends up
with a tree whose suite cannot collect:

FAIL src/agent-farm/__tests__/spec-146-phase-9-live-harness.test.ts
FAIL src/agent-farm/__tests__/spec-146-phase-9-porch-engine.test.ts
Error: Cannot find module '../../../../../porch-driver/dist/thread.js'
 imported from …/__tests__/helpers/porch-thread-engine.ts

That message is one of the two honest ones — it names the file — but it still describes a
path, not the action. The reader has to work out that the path belongs to another package and
that the build they just ran does not cover it.

CI is covered: #181 added a step for exactly this, filed when a fresh CI checkout failed the
same way. The same hole has existed locally the whole time and was invisible because everyone's
tree happened to already be built. That is the part worth fixing here — a green CI cannot tell
you your local build instructions are incomplete, because CI compensates for the gap rather than
exposing it.

So the ask grows a fourth point: the local build should either produce every artifact the suite
imports, or something should assert their presence by name before the suite runs. Either
closes all five messages at once, since all five share the precondition "the tree is not fully
built".

Verified in .builders/bugfix-196 on 2026-08-29: pnpm build in packages/codev produced
skeleton/ and dist/ and took the suite from 80 failures to 0, leaving exactly these two
collection errors; pnpm --filter @cluesmith/porch-driver build was needed separately.