Make the full verification battery safe to run concurrently

[taras]

Motivation

The repository's verification battery is logically independent, but it is not safe to run concurrently in one worktree. That forces local verification to serialize long Deno, Node, and Bun suites even though CI runs them in parallel on separate checkouts.

Owner ruling (2026-08-02): the entire applicable verification battery must be safe to launch concurrently in one worktree after setup. This is a repository rule, not an optimization that individual implementors may opt out of.

Reproduced failure

The current setup is:

deno task build:web
pnpm install

A concurrent local run of the pinned Deno suite with the Node and Bun suites then produced false failures such as:

ERR_MODULE_NOT_FOUND: Cannot find package '@effectionx/context-api'
ENOENT reading node_modules/@effectionx/converge

The same PR head is green in CI because the runtime jobs use separate checkouts.

The collision is structural:

• The root deno.json uses "nodeModulesDir": "auto".
• scripts/build-web-client.ts calls deno install --frozen in the repository root.
• deno task test executes scripts/tests/build-web-client.test.ts, which calls buildWebClient() repeatedly.
• That Deno install replaces the pnpm-shaped root node_modules while pnpm exec tsc, the Node suite, the Bun suite, lint tools, and spawned tsx processes resolve through it.
• buildWebClient() also temporarily patches the fixed path node_modules/@rjsf/validator-ajv8/package.json and writes the fixed generated path packages/web/generated/client-bundle.ts. Concurrent builders can therefore observe one another's temporary manifest state or race restoration/output writes.
• The Deno install follows the workspace CLI bin target and changes the tracked mode of packages/cli/src/node.ts from 100644 to 100755, so verification can leave a clean tree dirty even without another runtime running.

The first concurrent Node/Bun results are not product failures: they were caused by this shared-state race. Pinned Deno 2.9.1 completed independently with 320 passed and 0 failed.

Contract

After one documented setup, all applicable commands in the verification battery can be started concurrently from the same worktree:

deno task lint
deno task check
deno task test
deno task check:jsr
pnpm exec tsc --project tsconfig.node.json --noEmit
pnpm test:node
bun run test:bun
deno task xmd test packages/core/src --raw
(cd site && deno task check)
(cd site && deno task build)

The site commands are applicable when site/ changes, as today. Deno commands run with CI-pinned Deno 2.9.1 on PATH.

The invariant is stronger than "the commands usually pass together":

• No verification command may install into, remove, relink, patch, chmod, or clean the repository dependency tree or another command's owned paths.
• Temporary mutations are isolated per command/process; cleanup from one command cannot overwrite another command's state.
• Generated outputs needed by verification remain deterministic, but a fixed generated path must not become a cross-command race.
• A successful battery leaves tracked files byte-for-byte and mode-for-mode unchanged.
• Failures and cancellation also restore or remove only the invoking command's own temporary state.
• CI keeps its independent jobs; this issue fixes local composability rather than collapsing CI into one job.

Dependency-cache boundary

Owner ruling (2026-08-04): verification does not have a Deno global-cache purity requirement. The runtime-owned cache may gain or refresh locked npm/JSR package content and registry metadata while checks run. Verification must not fingerprint that cache or require it to remain byte-identical.

The protected repository state remains strict: verification cannot change node_modules, deno.lock, tracked content or modes, or another invocation's temporary state. Build and release phases retain the separate #304 guarantee: after preparation they run offline and leave both dependency layouts and the lock unchanged. verify:clean continues checking cache purity around those build/release phases, but not around the verification battery.

Repository rule

Update AGENTS.md so future changes preserve this invariant. The rule must say that the full verification battery is designed to run concurrently after setup and that tests/build helpers must not rely on repository-wide mutable state that conflicts with another check.

The rule supplements the existing command list; it does not weaken any required check or permit skipping a command.

Plan gate

This is a nontrivial mechanism change. Before code, submit a short plan that identifies:

1. the owner and lifetime of dependency layouts used by Deno, pnpm/Node, Bun, lint, and site checks;
2. how buildWebClient() stops mutating shared dependency/package paths while preserving the real-bundle and deterministic-output tests;
3. how generated bundle and site build outputs are isolated or coordinated without serializing the battery;
4. how tracked file mode/content cleanliness is enforced; and
5. the regression harness that launches the whole applicable battery concurrently and proves both results and post-run cleanliness.

Do not choose product behavior in the implementation plan. The product contract is the concurrent-safety invariant above; the mechanism remains open for review.

Acceptance criteria

• From a clean checkout after the documented setup, one command or documented harness launches the full applicable battery concurrently and every command passes.
• The regression fails against the current shared-node_modules behavior.
• Repeated runs are deterministic and leave no tracked content or mode changes, no node_modules changes, and no deno.lock changes; Deno global-cache additions are permitted.
• Failure/cancellation coverage proves temporary state is scoped to the invoking command.
• Existing CI jobs remain green with Deno 2.9.1, Node 22, and the pinned Bun version.
• AGENTS.md records the concurrent-verification rule.
• The setup and local verification documentation show the parallel invocation.

[taras]

Planner review of revised mechanism plan, revision 2 (2026-08-02): implementation remains paused pending one more revision.

Rulings:

• Keep the clean-checkout composability regression as a dedicated parallel CI job and deno task verify:clean, not inside deno task test. It owns installation and compilation and must run from a clean checkout.
• Keep the diagnosed findFreePort() bind race out of Make the full verification battery safe to run concurrently #279. Track it separately; do not widen either stage with that pre-existing defect.
• Stage 1 is part of Make the full verification battery safe to run concurrently #279 and must not use Closes #279; Stage 2 closes the issue after the concurrent harness, repository rule, and documentation land.

Required plan corrections before code:

1. The proposed NUL pipeline is not valid. git ls-files -z emits NUL-delimited paths, while git hash-object --stdin-paths consumes newline-delimited paths. A probe with a tracked filename containing a newline split the name and failed. Use a genuinely NUL-safe enumeration and hashing mechanism and add that filename as a regression; do not translate NUL to newline.
2. A pure build must override Deno automatic node-module management. Under Deno 2.9.1, deno eval --node-modules-dir=auto --cached-only with a cached npm import created node_modules/.deno before application code ran. A runtime assertion is therefore too late. Specify and empirically test a mode such as --node-modules-dir=manual that reads the prepared union tree but never creates or relinks it; an unprepared build must fail naming setup without changing the dependency tree.
3. Unique staging names prevent cross-invocation deletion but do not by themselves make cancellation clean. The staged write is promise-backed and can finish after halt cleanup. The normalizer must retain and observe any in-flight write before removing its own staged file. Test halt during a deliberately held write, then release it and prove the cancelled invocation leaves no late file while another invocation remains intact.
4. Do not append verify --no-site after separate tsc and Node-suite runs in Stage 2: verify already contains them and the sequence would make the new job the CI critical path. Stage 1 may use focused post-build probes while the harness does not exist; Stage 2 should replace the duplicate checks with setup -> pure build -> resolution probe -> verify --no-site.
5. Describe the composability job as additional runner cost and measure its final duration. Running in parallel does not prove it cannot extend the workflow critical path.

The workflow/spec coverage and the two-stage separation are otherwise directionally correct. PR #284 must remain draft and unchanged until the revised plan is approved and the currently active PR sequence permits implementation.

[taras]

Planner review of revision 3 (2026-08-02): two factual mechanism gaps remain; no code is authorized yet.

1. git diff --name-only -z does not emit both paths of a committed rename. In a scratch repository, renaming old to site/new produced only site/new from --name-only -z; git diff --name-status -z produced R100, old, and site/new. The current plan would therefore miss the site/ side of a committed rename out of that directory. Use a status-aware NUL parser for the branch diff, explicitly enable rename detection, consume both paths for rename/copy records, and add committed branch renames into and out of site/ to the regressions. Keep the separate status-aware parser for git status --porcelain=v1 -z, whose rename field order differs.

2. --node-modules-dir=none prevents local node_modules management but still allows Deno to fetch dependencies into the shared global cache. Deno 2.9.1 exposes --cached-only as the separate control for this. The plan must name deps/setup as owner of both the prepared union tree and the required global dependency cache, and make every pure build phase unable to fetch as well as unable to relink: the outer script, spawned bundler, and compile step use --cached-only or an empirically equivalent mechanism. Extend the four probes to prove a prepared build succeeds offline and an unprepared build changes neither dependency layout. If the custom setup diagnostic requires a dependency-free preflight wrapper before the cached graph loads, specify that rather than relying on a statement that cannot execute on a cold cache.

Revision 3 otherwise incorporates the prior rulings correctly. PR #284 remains draft at unchanged head a0702a9.

[taras]

Planner review of revision 4 (2026-08-02): two contained corrections remain; implementation stays paused.

1. Correct the exact worktree porcelain contract and command flags. git status --porcelain=v1 -z emits XY<space>new-path\0old-path\0 for a rename: the status and first path share the first NUL-delimited field. Revision 4 currently shows XY\0new-path\0old-path\0, which would drive the wrong parser. Parse the fixed status prefix from the first field, then consume the second field only for R/C. For the branch command, include explicit -M/--find-renames rather than relying on repository or user Git configuration; include -C if the promised copy integration test expects Git to produce C records.

2. --cached-only prevents dependency fetching but does not make the lockfile read-only. Add explicit --frozen to every pure build phase (outer build process, spawned bundler, and compile) so a local build cannot update deno.lock; also use --frozen on the deno install --entrypoint preparation step so setup refuses stale declarations instead of legitimizing them. Extend the purity probe to fingerprint deno.lock as well as both dependency layouts.

The new entrypoint-cache premise was probed against clean current main with pinned Deno 2.9.1: deno install --frozen --entrypoint scripts/build-web-client.ts packages/web/client/main.tsx packages/cli/src/compiled.ts succeeded even though the generated client module was absent. The staged-write, fingerprint, composability, offline-cache, and two-stage mechanisms are otherwise approved in revision 4. PR #284 remains draft at unchanged head a0702a9.

[taras]

Planner review of revision 5 (2026-08-02): approved.

Empirical review confirmed the corrected git status --porcelain=v1 -z record shape, explicit rename parsing, copy behavior (-M -C reports an addition for an unchanged source while --find-copies-harder produces C100), and the pinned Deno 2.9.1 bundle flag surface (--frozen, --no-remote, and manual node-modules mode; no --cached-only). PR #284 remains draft at unchanged head a0702a9.

Stage 1 may proceed only when the active one-PR sequence permits it. Run and report the listed dependency-layout/offline probes before retaining implementation changes; if any probe contradicts the plan, stop and return the measurements for a new ruling. Stage 1 remains Part of #279 and stops for review. Stage 2 is not started until Stage 1 merges and is the PR that closes #279.

[taras]

Retrospective audit of merged PR #284 — FAIL

PR #284 merged before final planner review. I audited final head 74684fc and merge commit de4f775 against the approved revision-5 plan and the current #279 contract. Stage 1 is not complete.

Required ordering ruling: land one focused Stage-1 repair PR before beginning Stage 2. Do not create a release tag until that repair merges; the current release compile is not frozen/offline. #279 remains open for Stage 2.

The repair must address all of these findings:

1. Preparation is not frozen at its entry boundary. deno task deps and deno task setup start with an unfrozen outer deno run. In a scratch worktree, adding an already-cached npm:picocolors@1.1.1 import to scripts/deps.ts made deno task deps add the top-level specifier to deno.lock; its later internal deno install --frozen then succeeded against the rewritten lock. Put --frozen on the outer task entry points and add a regression proving a stale graph fails without changing deno.lock.
2. The release compile bypasses the approved build invariant. .github/workflows/release.yml uses --node-modules-dir=none but omits --cached-only --frozen. Add both flags, update the release workflow regression and specs/release-process-spec.md, and exercise the release compile path in the offline/lock-purity proof.
3. The dependency-purity regression does not prove the approved contract. dependencyLayout() only lists top-level non-dot entries and symlink targets. It cannot see package-byte changes, modes, .deno content, the Deno cache, or deno.lock. Implement the revision-5 fingerprints for the complete prepared node_modules state, dependency content in the selected Deno cache, and deno.lock, and assert them around every build phase. Add the real cold-preflight integration with a scratch DENO_DIR and absent preparation; the current assertSideEffectFree([missing]) unit case does not test preflight.
4. One test invokes the wrong seam. scripts/tests/build-web-client.test.ts directly runs scripts/build-web-client.ts without --node-modules-dir=none --cached-only --frozen, and its comment still claims deno task build:web installs. Exercise the protected task or the exact production flags and remove the stale premise.
5. Two repository code rules were bypassed without an owner ruling. scripts/preflight.ts uses top-level await, despite the no-Promise/async/await rule; scripts/lib/prepared.ts uses an as cast instead of parsing/narrowing. Remove both violations. If the dependency-free preflight truly requires a code-rule exception, stop with a short mechanism plan for owner review rather than declaring the exception in source.

The final PR-head checks were green and there were no review threads; that does not cover these mechanisms. The merge-commit CI run was still in progress when audited. Run the pinned full battery and obtain complete green CI on the repair PR.

[taras]

Merge-commit CI run 30768675421 has now completed green. The retrospective FAIL and required Stage-1 repair are unchanged: the defects above are outside what those checks assert.

[taras]

Amendment to finding 1: apply the frozen entry-boundary regression to every new dependency-loading task that must not dirty the caller, including verify:clean, not only deps and setup. The demonstrated exploit is in deps; the invariant is task-entry-wide.

[taras]

Planner review of the Stage-1 repair plan — FAIL

No code is authorized yet. Sections 2, 4, and 5 are approved as written; the preparation/cache proof needs one revision.

1. The shared-cache fingerprint may not be replaced by an offline run. Network-unreachable execution proves that a phase does not need a fetch. It does not detect writes made to dependency-cache content from already-local inputs, so it is not stronger than the revision-5 before/after invariant. Keep the offline run as a supplement. Give the clean-checkout/composability invocation its own scratch DENO_DIR, prepare that cache once during the setup step, fingerprint its dependency content, and compare it around build:web, build, and the release compile. This removes the shared-cache race instead of weakening the assertion. If a different isolated mechanism is proposed, demonstrate that it fails on a phase which mutates existing cached dependency content without using the network.
2. The proposed prepared-tree fingerprint is still too shallow. Listing non-store paths plus top-level .deno/.pnpm entries detects additions, removals, and relinks, but not byte or mode changes inside an existing store entry. Make the full verification battery safe to run concurrently #279 explicitly covers patching and chmod, and the failed audit called out both. The regression must include content and executable mode for the complete prepared dependency state, or an empirically equivalent mechanism that fails on representative byte and mode mutations inside existing .deno and .pnpm packages. Measure and report the real cost; do not infer that 152,116 paths is prohibitive.
3. verify:clean is not an installer at its outer boundary. Only deps and setup may fetch/manage the caller's dependency state. The current verify:clean task uses automatic node-module management before its clone exists; adding only --frozen leaves that violation intact. Its outer entry must be unable to fetch or manage the caller's node_modules (the established none + cached-only + frozen mechanism, or measured equivalent), while the setup inside its isolated clone owns that clone's tree and per-invocation cache. Extend the stale-lock/task-entry regression to cover deps, setup, and verify:clean, not only deps.

Revise the plan and stop again before code. Keep the existing release block and one-repair-PR ordering.

[taras]

Planner review of Stage-1 repair plan revision 2 — FAIL

The full prepared node_modules fingerprint, isolated scratch-cache ownership, outer verify:clean flags, three-entry stale-lock matrix, and carried sections are approved. One empirical correction remains before code.

Revision 2 fingerprints every path under DENO_DIR and requires the whole cache to remain byte-identical. That is not the approved dependency-content invariant and fails during normal offline execution. With pinned Deno 2.9.1, I ran deno install --entrypoint for a dependency-free local script in a fresh scratch DENO_DIR, fingerprinted it, then ran the script with --cached-only. The run fetched no dependency, but it created v8_code_cache_v2, changed dep_analysis_cache_v2-wal, and created analysis/code-cache WAL and SHM files. Entrypoint preparation therefore does not make the complete cache immutable.

Revise the cache fingerprint to cover dependency-content roots only (empirically enumerate the Deno 2.9.1 roots, including the relevant npm/ and remote/JSR content) and explicitly exclude derived analysis, generated-code, V8, WAL, and SHM state. Preserve the scratch DENO_DIR, offline supplement, and before/after comparison. The mutation regressions must alter bytes inside actual cached dependency content and prove that the scoped fingerprint moves; normal offline execution that changes only derived caches must not move it.

Also make every stale-lock case assert the expected frozen-lock diagnostic in addition to non-zero exit and byte-identical deno.lock, particularly verify:clean, so a missing cache cannot satisfy the regression for the wrong reason.

Revise the plan and stop again before code. The release block and one-repair-PR ordering remain in force.

[taras]

Planner review of Stage-1 repair plan revision 3 — FAIL

The derived-cache exclusion and stale-lock diagnostic correction are approved. One provenance correction remains; no code is authorized yet.

The content-root inclusion list was classified from the machine's long-lived shared cache, which contains material from multiple Deno versions and tools. That is not evidence that a fresh pinned-2.9.1 scratch cache created by this repository's setup has the same roots. Empirically, pinned deno info --json identifies remote/ as modulesCache, npm/ as npmCache, gen/ as the derived typescriptCache, and registries/ as registryCache; it does not identify deps/ or deno_x_cache/. The latter contains standalone deno x environments, and neither root has yet been shown to be created by this repository's setup. Revision 3 nevertheless requires every listed root to exist and be non-empty, so the test can fail on absent legacy/tool roots before it proves build purity.

Before code, run the exact pinned deno task setup against a fresh per-invocation scratch DENO_DIR and report both deno info --json for that directory and the roots setup actually creates. Revise the fingerprint to derive or validate dependency-content roots from that pinned runtime/scratch preparation, not from the shared cache. Include absence/presence in the baseline so a later content root cannot appear unnoticed, but require non-empty only for roots the repository's prepared graph actually uses. Do not include deps/ or deno_x_cache/ unless the fresh setup proves they are active dependency-content roots for this workflow.

Preserve the full node_modules content/mode fingerprint, scratch ownership, offline supplement, derived-cache churn regression, three-entry frozen diagnostic matrix, release block, and one-repair-PR ordering. Revise the plan and stop again before code.

[taras]

Planner review of Stage-1 repair plan revision 4 — FAIL

The fresh-cache provenance experiment resolves the previous finding. Ruling on cost: keep the baseline plus one fingerprint after each of the three phases. A single final fingerprint could miss one phase mutating dependency state and a later phase restoring it; per-phase coverage is part of the invariant, not merely attribution. The measured two-to-three-minute composability job remains below the current 5m17s critical path and is acceptable, subject to reporting the final CI duration.

One carried contract item is missing from the current plan text: the fingerprint in §2 explicitly covers node_modules and the three Deno cache roots, but no longer includes deno.lock. The retrospective audit and approved revision-5 proof require the lock digest before and after every build phase. The stale-entry tests in §3 do not replace that phase-level assertion.

Revise §2 so each of the four snapshots also includes the clone's deno.lock bytes/digest, and add a focused regression proving a lock-byte mutation moves the prepared-state fingerprint. Preserve all other revision-4 mechanisms and the four-snapshot sequence, then stop once more before code.