diff --git a/.gitattributes b/.gitattributes index a6663fd5..c77f36cc 100644 --- a/.gitattributes +++ b/.gitattributes @@ -40,6 +40,8 @@ scripts/tests/test_host_gate.py text eol=lf scripts/tests/test_bootstrap.py text eol=lf scripts/build_dist.py text eol=lf scripts/tests/test_build_dist.py text eol=lf +scripts/carry.py text eol=lf +scripts/tests/test_carry.py text eol=lf scripts/skills_install.py text eol=lf scripts/tests/test_skills_install.py text eol=lf diff --git a/AUDIT.md b/AUDIT.md index a3d854fa..55908f26 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -81,6 +81,7 @@ A check with `intentRef`/`workflowRef` points at the prose section that owns the - **hugo** - the build fails on a generator warning, the URL-parity gate asserts a length floor before comparing, the rendered output is untracked, the generator is pinned by version and checksum and declared once, a vendored tree records its upstream ref, and the deploy asserts what the host serves (the release id and the environment). Retention is bounded by a declared count with one side recorded as owning the prune, which is the deploy where its credential can observe the destination and the host where that credential is confined write-only, so grade which shape the repo uses rather than looking for a prune step. Deploy credentials are per-environment, which `spec/secrets.json` cannot express, so a clean **repo-setup** verdict says nothing about whether the environments are configured. - **branch-model** - `main` and `develop` both exist and are protected, and the live rulesets match [`repo-config/*.json`][repo-config] by normalized diff (below). - **carried-scope** - the repo carries no file the hub hosts rather than carries. The set is derived, not listed: the hub's git-tracked paths minus the [`spec/files.json`][files] baseline, so a file dropped from the manifest starts being reported on the next run with no retirement list to remember to edit. The remedy is the opposite of every other file finding, a **deletion**, since the repo reaches the hub's copy per [GOVERNANCE.md "Hub-Hosted Tooling"][governance-hub-hosted-tooling]. The match is on path alone, so a hit is a candidate and not a verdict: a repo's own content at a path the hub also uses matches while carrying nothing of the hub's, which the first fleet run showed twice, a KiCad tooling doc at `scripts/README.md` and per-repo formatting hooks at `.husky/pre-commit`. A [`spec/divergences.json`][divergences] `gaps` disposition decides which case a hit is, so only `retire` asserts a deletion, `accepted` closes a collision or a repo-owned file, and an untriaged hit is read before it is acted on. +- **verbatim-tree** - every applicable `trees[]` declaration in [`spec/files.json`][files] owns its target tree. The audit reports missing files as letter findings, stale or modified bytes as drift, and extra files under a pruned target as drift. An unreadable or truncated repository tree is undecided and produces drift rather than a clean result. - **repo-setup** - every required secret for the repo's publish mechanisms is configured, and no forbidden secret is present (per [`spec/secrets.json`][secrets]). - **linter-parity** - one config per linter (`.markdownlint-cli2.jsonc`, `cspell.json`, ruff/pyright, editorconfig/csharpier, actionlint) drives the editor extension, the CLI, and CI, and CI runs each. - **recurring-violations** - comments concise and non-narrative, ASCII only (no em-dash, no smart quotes), US spelling, line endings per `.editorconfig`. These are frequent regressions, so this dimension is high priority and always runs, and each check is grep-able (see below). diff --git a/RESYNC.md b/RESYNC.md index 841b52b2..ef35dff7 100644 --- a/RESYNC.md +++ b/RESYNC.md @@ -72,6 +72,7 @@ The order is load-bearing. Each step below either changes the rules the later st 1. **The instruction set first.** `AGENTS.md` and `GOVERNANCE.md` verbatim sections, then `CODESTYLE.md` and `WORKFLOW.md`. `AGENTS.md`'s skill-dependency pointer paragraph, naming `scripts/skills_install.py` and where the fleet's Skills live, carries as one more verbatim unit in this same step, not a separate pass. These are the rules for producing every other file, so carrying them last means everything touched beforehand was judged against the previous revision. This is the same closing-window shape as [`STANDUP.md`][standup] section 1A, and the cost of getting it wrong is rework proportional to how much was changed first. **Run the `carried-instruction-file-guard` skill's distinctive-phrase probe before any verbatim re-vendor in this step, every time, without exception.** A diff that looks routine is exactly the shape the AGENTS.md-overwrite incident took, so the probe is not weighed against how routine the request sounds, it runs regardless. `resync-a-repo` packages this whole step, and the rest of this section, as a Skill for a hub-context session driving the resync. 2. **Deletions second, before any re-vendor.** A `hub-only:` finding names a file the hub hosts rather than carries, and its remedy removes the file. Doing it after the re-vendors means refreshing a copy that is about to be deleted, which is wasted work that also reads as a deliberate update in the diff. See section 4, which is the whole of what deletion means here. 3. **Verbatim re-vendors.** Copy the current hub canonical down, whole file or the one named `## heading` region. A finding classified **stale** matches a past hub revision and needs no judgment. One classified **modified** matches no revision, so the repository changed fixed content and the change is read before it is overwritten, since it may be an improvement the hub should adopt instead. + Run `python3 scripts/carry.py check --target /path/to/worktree` for manifest-owned trees. Read each modified-file diff and every extra path before running the same command with `apply`. The tree declaration supplies the prune authority, so the carried-instruction-file guard does not apply to this fully owned content. 4. **Interface workflows.** Honor the named contract (required jobs, the ruleset-bound check name, the artifact-name handoff) rather than copying bytes. The body is the repository's own. 5. **Settings, rulesets, and secrets.** Run the hub's script against the repository by name, never a carried copy: `repo-config/configure.sh check / release|operational`, then `apply` for what it reports. Pass the model explicitly rather than relying on the registry lookup. 6. **Intent files last, and by hand.** See section 5, which states why these carry no mechanical signal at all. diff --git a/STANDUP.md b/STANDUP.md index df9957e1..3cf247cc 100644 --- a/STANDUP.md +++ b/STANDUP.md @@ -140,6 +140,15 @@ Capture the source, verify the capture **against the source**, and hold the veri ## 2. Carry the Baseline Files +After the baseline files are present, apply every applicable whole-tree declaration from the fetched hub checkout. The carry tool resolves the repository through the registry and refuses a shared or dirty target worktree. + +```shell +python3 scripts/carry.py check --target /path/to/worktree +python3 scripts/carry.py apply --target /path/to/worktree +``` + +Read every extra-path report before `apply` prunes it. Finish with another `check`. It uses the same comparison as `apply` and must report clean. + Copy every [`spec/files.json`][files] entry whose `appliesTo` matches the repo's **selector set**, **adapted, not cloned**. The selector set is the repo's `types` plus its `workflowModel`, `releaseTrigger`, and `consumerModel`, so filtering on type alone silently drops the entries a non-type selector carries ([`spec/scope-model.md`][scope-model] defines the four namespaces and how they resolve). The prose files (`CODESTYLE.md`, `README.md`, and the like) describe the repo's own toolchain, so adapt them to reality rather than propagating template specifics verbatim (see the "Adapt before propagating" callout in [`CODESTYLE.md`][codestyle], since a verbatim copy that misdescribes the repo is rejected in review). The baseline covers `WORKFLOW.md`, `version.json`, the two rulesets, `.github/dependabot.yml`, `.editorconfig`, `.gitattributes`, `host-tools.json`, the linter configs, and the per-type files (`.vscode/tasks.json` from the language's snippet, `codecov.yml`, `.dockerignore`, `Docker/README.md`). **Every repo carries `repo-config/main.json`**, and only the `develop` payload varies by workflow model: `repo-config/develop.json` for a release repo, `repo-config/operational/develop.json` for an operational one. Carry `AGENTS.md`'s skill-dependency pointer paragraph, the one naming `scripts/skills_install.py` and where the fleet's Skills live, as one more verbatim unit in this same step, not a separate pass. It reads like boilerplate next to the surrounding text a new repo adapts to describe itself, and a repo that carries `AGENTS.md` without it stands up with no path to the fleet's Skills at all. `RESYNC.md` carries the identical instruction for a repo already stood up, so the two procedures agree on what belongs in every copy. diff --git a/scripts/README.md b/scripts/README.md index 6dd4363f..3f89ccb3 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -200,6 +200,17 @@ Regenerates [`.github/skills/`][github-skills-dist] and [`.claude-plugin/fleet-s `--check` is the read-only mode: it exits `1` when either generated tree differs from `.agents/skills/`, comparing a digest over every file rather than a file count or timestamp. CI runs `--check` rather than trusting a contributor to have run the generator, the same reason `spec/audit.py` exists rather than trusting a hand-carried file. +## `carry.py` + +Copies manifest-owned trees from a freshly fetched ProjectTemplate `main` checkout into an isolated downstream feature worktree. `check` reports the canonical commit, repository identity, applicable declarations, path differences, and tree digests without changing the target. `apply` reports each write or removal and repeats the comparison before it succeeds. + +```shell +python3 scripts/carry.py check PhotoCleaner --target /path/to/worktree +python3 scripts/carry.py apply PhotoCleaner --target /path/to/worktree +``` + +The tool validates the registry identity, target origin, feature-branch worktree, development-branch ancestry, unrelated changes, root containment, and symlink-free trees. A pruned declaration authorizes removal only beneath its target root. + ## `skills_install.py` Installs the fleet's Skills for the current machine, cross-platform and idempotent, mirroring [`host-setup/agent-safety/install.py`][agent-safety-install]'s shape: `skills_install.sh` and `skills_install.ps1` are thin wrappers that locate a Python 3 interpreter and hand off, so every OS runs one tested code path. Two independent things happen on a run, since the three tools this fleet targets discover skills differently: `.agents/skills/` is materialized (not symlinked) to `$HOME/.agents/skills/`, so Codex and opencode's global scan covers every repo on the machine rather than only the one that happens to be open, and this repo's marketplace is registered with the `claude` CLI (`claude plugin marketplace add`, `claude plugin install`) so Claude Code loads the same content the other two read directly. The marketplace/plugin registration goes through the `claude` CLI's own commands rather than writing its internal `known_marketplaces.json` by hand, because that file's shape is the CLI's state, not a documented contract, and a hand-written copy risks drifting from what the CLI expects on its next release. diff --git a/scripts/carry.py b/scripts/carry.py new file mode 100755 index 00000000..abdaab2a --- /dev/null +++ b/scripts/carry.py @@ -0,0 +1,467 @@ +#!/usr/bin/env python3 +"""Check or apply manifest-owned trees from the hub to a fleet worktree.""" + +import argparse +import fnmatch +import hashlib +import json +import os +import pathlib +import subprocess +import sys +from dataclasses import dataclass +from typing import Any + +ROOT = pathlib.Path(__file__).resolve().parent.parent + + +class CarryError(RuntimeError): + """A state that prevents a safe carry decision.""" + + +@dataclass(frozen=True) +class Inventory: + files: dict[str, bytes] + directories: frozenset[str] + digest: str + + +def load_json(path: pathlib.Path) -> Any: + try: + return json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise CarryError(f"cannot read {path}: {exc}") from exc + + +def relative_root(root: pathlib.Path, value: str) -> pathlib.Path: + declared = pathlib.PurePosixPath(value) + if declared == pathlib.PurePosixPath(".") or declared.is_absolute() or ".." in declared.parts: + raise CarryError( + f"path must be repository-relative and below the repository root without '..': {value}" + ) + resolved_root = root.resolve() + candidate = resolved_root / value + resolved_candidate = candidate.resolve(strict=False) + try: + resolved_candidate.relative_to(resolved_root) + except ValueError as exc: + raise CarryError(f"path escapes repository root: {value}") from exc + current = resolved_root + for part in pathlib.PurePosixPath(value).parts: + current /= part + if current.is_symlink(): + raise CarryError(f"symlink is not allowed: {current}") + return candidate + + +def included(path: str, patterns: list[str]) -> bool: + return any( + pattern == "**/*" + or fnmatch.fnmatchcase(path, pattern) + or pathlib.PurePosixPath(path).match(pattern) + for pattern in patterns + ) + + +def inventory(root: pathlib.Path, patterns: list[str]) -> Inventory: + files: dict[str, bytes] = {} + directories: set[str] = set() + if root.is_symlink(): + raise CarryError(f"symlink is not allowed: {root}") + if not root.exists(): + raise CarryError(f"tree root does not exist: {root}") + if not root.is_dir(): + raise CarryError(f"tree root is not a directory: {root}") + for current, dirnames, filenames in os.walk(root, followlinks=False): + current_path = pathlib.Path(current) + for name in [*dirnames, *filenames]: + path = current_path / name + if path.is_symlink(): + raise CarryError(f"symlink is not allowed: {path}") + relative_dir = current_path.relative_to(root).as_posix() + if ( + relative_dir != "." + and not dirnames + and not filenames + and included(relative_dir + "/placeholder", patterns) + ): + directories.add(relative_dir) + for name in filenames: + path = current_path / name + relative = path.relative_to(root).as_posix() + if included(relative, patterns): + try: + files[relative] = path.read_bytes() + except OSError as exc: + raise CarryError(f"cannot read {path}: {exc}") from exc + digest = hashlib.sha256() + for relative in sorted(directories): + digest.update(b"directory\0") + digest.update(relative.encode()) + digest.update(b"\0") + for relative, content in sorted(files.items()): + digest.update(b"file\0") + digest.update(relative.encode()) + digest.update(b"\0") + digest.update(content) + digest.update(b"\0") + return Inventory(files, frozenset(directories), digest.hexdigest()) + + +def compare(source: Inventory, target: Inventory | None) -> dict[str, Any]: + target_files = {} if target is None else target.files + target_directories = frozenset() if target is None else target.directories + return { + "missing": sorted(set(source.files) - set(target_files)), + "missingDirectories": sorted(source.directories - target_directories), + "modified": sorted( + path + for path in set(source.files) & set(target_files) + if source.files[path] != target_files[path] + ), + "extra": sorted(set(target_files) - set(source.files)), + "extraDirectories": sorted(target_directories - source.directories), + "sourceDigest": source.digest, + "targetDigest": None if target is None else target.digest, + "missingRoot": target is None, + } + + +def apply_tree( + source: Inventory, + target_root: pathlib.Path, + repository_root: pathlib.Path, + result: dict[str, Any], +) -> list[str]: + changes: list[str] = [] + required_directories = set(source.directories) + for relative in [*source.files, *source.directories]: + required_directories.update( + str(parent) for parent in pathlib.PurePosixPath(relative).parents if str(parent) != "." + ) + + def remove_empty_ancestors(directory: pathlib.Path) -> None: + while directory != target_root and directory.exists() and not any(directory.iterdir()): + relative = directory.relative_to(target_root).as_posix() + if relative in required_directories: + return + directory.rmdir() + changes.append(f"remove {directory.relative_to(repository_root)}") + directory = directory.parent + + created_roots = [] + current = target_root + while current != repository_root and not current.exists(): + created_roots.append(current) + current = current.parent + target_root.mkdir(parents=True, exist_ok=True) + changes.extend( + f"create {directory.relative_to(repository_root)}" for directory in reversed(created_roots) + ) + for relative in result["missingDirectories"]: + destination = target_root / relative + destination.mkdir(parents=True, exist_ok=True) + changes.append(f"create {destination.relative_to(repository_root)}") + for relative in [*result["missing"], *result["modified"]]: + destination = target_root / relative + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_bytes(source.files[relative]) + changes.append(f"write {destination.relative_to(repository_root)}") + for relative in result["extra"]: + destination = target_root / relative + destination.unlink() + changes.append(f"remove {destination.relative_to(repository_root)}") + remove_empty_ancestors(destination.parent) + for relative in sorted( + result["extraDirectories"], + key=lambda value: len(pathlib.PurePosixPath(value).parts), + reverse=True, + ): + directory = target_root / relative + if directory.exists() and not any(directory.iterdir()): + directory.rmdir() + changes.append(f"remove {directory.relative_to(repository_root)}") + remove_empty_ancestors(directory.parent) + return changes + + +def git(root: pathlib.Path, *args: str) -> str: + result = subprocess.run(["git", *args], cwd=root, capture_output=True, text=True, check=False) + if result.returncode != 0: + raise CarryError(result.stderr.strip() or f"git {' '.join(args)} failed") + return result.stdout.strip() + + +def git_is_ancestor(root: pathlib.Path, ancestor: str, descendant: str) -> bool: + result = subprocess.run( + ["git", "merge-base", "--is-ancestor", ancestor, descendant], + cwd=root, + capture_output=True, + text=True, + check=False, + ) + if result.returncode not in (0, 1): + raise CarryError(result.stderr.strip() or "git merge-base --is-ancestor failed") + return result.returncode == 0 + + +def git_status_paths(root: pathlib.Path) -> list[str]: + result = subprocess.run( + ["git", "status", "--porcelain=v1", "-z", "--untracked-files=all"], + cwd=root, + capture_output=True, + check=False, + ) + if result.returncode != 0: + raise CarryError(os.fsdecode(result.stderr).strip() or "git status failed") + fields = result.stdout.split(b"\0") + if fields and not fields[-1]: + fields.pop() + paths: list[str] = [] + index = 0 + while index < len(fields): + entry = fields[index] + if len(entry) < 4 or entry[2:3] != b" ": + raise CarryError("git status returned malformed porcelain output") + paths.append(os.fsdecode(entry[3:])) + if b"R" in entry[:2] or b"C" in entry[:2]: + index += 1 + if index >= len(fields): + raise CarryError("git status returned an incomplete rename or copy") + paths.append(os.fsdecode(fields[index])) + index += 1 + return paths + + +def normalized_origin(value: str) -> str: + value = value.strip().rstrip("/").removesuffix(".git") + if value.startswith("git@github.com:"): + return "https://github.com/" + value.removeprefix("git@github.com:") + if value.startswith("ssh://git@github.com/"): + return "https://github.com/" + value.removeprefix("ssh://git@github.com/") + return value + + +def verify_hub(hub: pathlib.Path, registry: dict[str, Any]) -> str: + entry = resolve_repo("ProjectTemplate", registry) + origin = normalized_origin(git(hub, "config", "--get", "remote.origin.url")) + if origin != normalized_origin(entry["url"]): + raise CarryError("hub origin does not match the ProjectTemplate registry entry") + git(hub, "fetch", "origin", "main") + head = git(hub, "rev-parse", "HEAD") + if head != git(hub, "rev-parse", "origin/main"): + raise CarryError("hub checkout is not at freshly fetched origin/main") + if git(hub, "status", "--porcelain"): + raise CarryError("hub checkout has local changes") + return head + + +def applicable(selector: str | list[str], values: set[str]) -> bool: + if selector == "*": + return True + tokens = selector if isinstance(selector, list) else [selector] + return bool(set(tokens) & values) + + +def validate_declarations(declarations: list[Any], hub: pathlib.Path) -> None: + required = {"source", "target", "fidelity", "appliesTo", "include", "prune"} + allowed = required | {"allowHubTarget"} + for index, left in enumerate(declarations): + if not isinstance(left, dict): + raise CarryError(f"tree declaration must be an object: {left!r}") + missing = required - set(left) + unknown = set(left) - allowed + if missing: + raise CarryError(f"tree declaration is missing: {', '.join(sorted(missing))}") + if unknown: + raise CarryError(f"tree declaration has unknown fields: {', '.join(sorted(unknown))}") + if not isinstance(left["source"], str) or not left["source"]: + raise CarryError("tree declaration source must be a non-empty string") + if not isinstance(left["target"], str) or not left["target"]: + raise CarryError("tree declaration target must be a non-empty string") + if left.get("fidelity") != "verbatim-tree": + raise CarryError(f"tree declaration has unsupported fidelity: {left.get('fidelity')}") + selector = left["appliesTo"] + if not ( + isinstance(selector, str) + or ( + isinstance(selector, list) + and selector + and all(isinstance(item, str) for item in selector) + ) + ): + raise CarryError( + "tree declaration appliesTo must be a string or non-empty string array" + ) + include = left["include"] + if not ( + isinstance(include, list) + and include + and all(isinstance(pattern, str) and pattern for pattern in include) + ): + raise CarryError("tree declaration include must be a non-empty string array") + if not isinstance(left["prune"], bool): + raise CarryError("tree declaration prune must be a boolean") + if "allowHubTarget" in left and not isinstance(left["allowHubTarget"], bool): + raise CarryError("tree declaration allowHubTarget must be a boolean") + relative_root(hub, left["source"]) + relative_root(hub, left["target"]) + for right in declarations[index + 1 :]: + left_target = pathlib.PurePosixPath(left["target"]) + right_target = pathlib.PurePosixPath(right["target"]) + overlaps = ( + left_target == right_target + or left_target in right_target.parents + or right_target in left_target.parents + ) + if overlaps: + raise CarryError( + f"overlapping tree declarations have conflicting ownership: {left_target} and {right_target}" + ) + + +def resolve_repo(name: str, registry: dict[str, Any]) -> dict[str, Any]: + matches = [entry for entry in registry.get("repos", []) if entry.get("name") == name] + if len(matches) != 1: + raise CarryError(f"repository is not uniquely registered: {name}") + return matches[0] + + +def verify_target( + target: pathlib.Path, entry: dict[str, Any], owned_roots: list[pathlib.Path] +) -> None: + top = pathlib.Path(git(target, "rev-parse", "--show-toplevel")).resolve() + if top != target.resolve(): + raise CarryError(f"target must name the repository root: {target}") + if normalized_origin(git(target, "config", "--get", "remote.origin.url")) != normalized_origin( + entry["url"] + ): + raise CarryError("target origin does not match the registry entry") + branch = git(target, "branch", "--show-current") + if not branch or branch in {"main", "develop"}: + raise CarryError("target must be an isolated feature-branch worktree") + if not (target / ".git").is_file(): + raise CarryError("target must be a linked worktree, not the primary checkout") + git(target, "fetch", "origin", "develop") + if not git_is_ancestor(target, "origin/develop", "HEAD"): + raise CarryError("target branch must contain the current origin/develop head") + worktree_rows = git(target, "worktree", "list", "--porcelain").splitlines() + if sum(row == f"worktree {target.resolve()}" for row in worktree_rows) != 1: + raise CarryError("target is not a registered git worktree") + dirty = [] + for relative in git_status_paths(target): + path = (target / relative).resolve(strict=False) + if not any(path == root or root in path.parents for root in owned_roots): + dirty.append(relative) + if dirty: + raise CarryError(f"target has unrelated changes: {', '.join(sorted(dirty))}") + + +def run(mode: str, name: str, target: pathlib.Path, hub: pathlib.Path = ROOT) -> int: + registry = load_json(hub / "registry/repos.json") + manifest = load_json(hub / "spec/files.json") + hub_commit = verify_hub(hub, registry) + entry = resolve_repo(name, registry) + defaults = registry.get("defaults", {}) + selectors = set(entry.get("types", [])) + selectors.add(entry.get("workflowModel") or defaults.get("workflowModel") or "release") + selectors.add(entry.get("releaseTrigger") or defaults.get("releaseTrigger") or "two-phase") + if entry.get("consumerModel"): + selectors.add(entry["consumerModel"]) + all_declarations = manifest.get("trees") + if not isinstance(all_declarations, list): + raise CarryError("manifest trees must be an array") + validate_declarations(all_declarations, hub) + declarations = [ + declaration + for declaration in all_declarations + if applicable(declaration.get("appliesTo", "*"), selectors) + ] + if name == "ProjectTemplate" and any( + not item.get("allowHubTarget", False) for item in declarations + ): + raise CarryError("a declaration does not allow ProjectTemplate as its target") + owned_roots = [relative_root(target, item["target"]) for item in declarations] + verify_target(target, entry, owned_roots) + print(f"hubCommit: {hub_commit}") + print(f"repository: {name}") + print(f"types: {','.join(entry.get('types', []))}") + print(f"declarations: {len(declarations)}") + clean = True + for declaration in declarations: + source_root = relative_root(hub, declaration["source"]) + target_root = relative_root(target, declaration["target"]) + source = inventory(source_root, declaration["include"]) + target_patterns = ["**/*"] if declaration.get("prune") else declaration["include"] + target_inventory = inventory(target_root, target_patterns) if target_root.exists() else None + result = compare(source, target_inventory) + if not declaration.get("prune"): + result["extra"] = [] + result["extraDirectories"] = [] + print( + json.dumps( + {"source": declaration["source"], "target": declaration["target"], **result}, + sort_keys=True, + ) + ) + declaration_clean = ( + not any( + result[key] + for key in ( + "missing", + "missingDirectories", + "modified", + "extra", + "extraDirectories", + ) + ) + and not result["missingRoot"] + ) + clean = clean and declaration_clean + if mode == "apply" and not declaration_clean: + for change in apply_tree(source, target_root, target, result): + print(change) + final = compare(source, inventory(target_root, target_patterns)) + print( + json.dumps( + { + "source": declaration["source"], + "target": declaration["target"], + "postApply": True, + **final, + }, + sort_keys=True, + ) + ) + if ( + any( + final[key] + for key in ( + "missing", + "missingDirectories", + "modified", + "extra", + "extraDirectories", + ) + ) + or final["missingRoot"] + ): + raise CarryError(f"post-apply comparison failed for {declaration['target']}") + return 0 if mode == "apply" or clean else 1 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("mode", choices=("check", "apply")) + parser.add_argument("repository") + parser.add_argument("--target", required=True, type=pathlib.Path) + args = parser.parse_args() + try: + return run(args.mode, args.repository, args.target.resolve()) + except CarryError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/tests/test_carry.py b/scripts/tests/test_carry.py new file mode 100755 index 00000000..97a04b79 --- /dev/null +++ b/scripts/tests/test_carry.py @@ -0,0 +1,314 @@ +#!/usr/bin/env python3 +"""Exercise carry.py's manifest inventory, safety checks, and apply behavior.""" + +from __future__ import annotations + +import pathlib +import subprocess +import sys +import tempfile +import unittest + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent)) +import carry + + +class CarryInventoryTests(unittest.TestCase): + def test_compare_reports_missing_modified_extra_and_directories(self) -> None: + source = carry.Inventory( + {"same.txt": b"same", "changed.txt": b"new", "missing.txt": b"missing"}, + frozenset({"empty", "nested"}), + "source", + ) + target = carry.Inventory( + {"same.txt": b"same", "changed.txt": b"old", "extra.txt": b"extra"}, + frozenset({"nested", "extra-dir"}), + "target", + ) + + result = carry.compare(source, target) + + self.assertEqual(result["missing"], ["missing.txt"]) + self.assertEqual(result["modified"], ["changed.txt"]) + self.assertEqual(result["extra"], ["extra.txt"]) + self.assertEqual(result["missingDirectories"], ["empty"]) + self.assertEqual(result["extraDirectories"], ["extra-dir"]) + + def test_empty_source_requires_target_root(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + source = root / "source" + source.mkdir() + + result = carry.compare(carry.inventory(source, ["**/*"]), None) + + self.assertTrue(result["missingRoot"]) + self.assertEqual(result["missing"], []) + + def test_inventory_detects_same_size_modification(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + left = root / "left" + right = root / "right" + left.mkdir() + right.mkdir() + (left / "value.txt").write_bytes(b"left") + (right / "value.txt").write_bytes(b"rite") + + result = carry.compare( + carry.inventory(left, ["**/*"]), carry.inventory(right, ["**/*"]) + ) + + self.assertEqual(result["modified"], ["value.txt"]) + + def test_inventory_digest_includes_empty_directories(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + left = root / "left" + right = root / "right" + (left / "left-empty").mkdir(parents=True) + (right / "right-empty").mkdir(parents=True) + + left_inventory = carry.inventory(left, ["**/*"]) + right_inventory = carry.inventory(right, ["**/*"]) + + self.assertNotEqual(left_inventory.digest, right_inventory.digest) + + def test_inventory_rejects_source_and_target_symlinks(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + tree = root / "tree" + tree.mkdir() + (tree / "real.txt").write_text("real", encoding="utf-8") + (tree / "link.txt").symlink_to("real.txt") + + with self.assertRaisesRegex(carry.CarryError, "symlink is not allowed"): + carry.inventory(tree, ["**/*"]) + + def test_inventory_read_failure_is_not_clean(self) -> None: + with tempfile.TemporaryDirectory() as temp: + missing = pathlib.Path(temp) / "missing" + with self.assertRaisesRegex(carry.CarryError, "does not exist"): + carry.inventory(missing, ["**/*"]) + + def test_apply_is_idempotent_and_preserves_unrelated_paths(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + source_root = root / "source" + target_root = root / "repo" / "owned" + unrelated = root / "repo" / "keep.txt" + source_root.mkdir() + (source_root / "empty").mkdir() + target_root.mkdir(parents=True) + (source_root / "current.txt").write_text("current", encoding="utf-8") + (target_root / "retired.txt").write_text("retired", encoding="utf-8") + unrelated.write_text("keep", encoding="utf-8") + source = carry.inventory(source_root, ["**/*"]) + first = carry.compare(source, carry.inventory(target_root, ["**/*"])) + + changes = carry.apply_tree(source, target_root, root / "repo", first) + final = carry.compare(source, carry.inventory(target_root, ["**/*"])) + second_changes = carry.apply_tree(source, target_root, root / "repo", final) + unrelated_content = unrelated.read_text(encoding="utf-8") + empty_directory_exists = (target_root / "empty").is_dir() + + self.assertEqual( + changes, + ["create owned/empty", "write owned/current.txt", "remove owned/retired.txt"], + ) + self.assertEqual(second_changes, []) + self.assertEqual(unrelated_content, "keep") + self.assertTrue(empty_directory_exists) + + def test_apply_reports_empty_target_root_creation(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + repository = root / "repo" + source_root = root / "source" + repository.mkdir() + source_root.mkdir() + source = carry.inventory(source_root, ["**/*"]) + target_root = repository / "owned" + + changes = carry.apply_tree(source, target_root, repository, carry.compare(source, None)) + + self.assertEqual(changes, ["create owned"]) + + def test_pruned_target_with_narrow_include_keeps_structural_directories(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + source_root = root / "source" + target_root = root / "target" + (source_root / "nested").mkdir(parents=True) + (target_root / "nested").mkdir(parents=True) + (source_root / "nested/value.txt").write_text("same", encoding="utf-8") + (target_root / "nested/value.txt").write_text("same", encoding="utf-8") + + result = carry.compare( + carry.inventory(source_root, ["*.txt"]), + carry.inventory(target_root, ["**/*"]), + ) + + self.assertEqual(result["extraDirectories"], []) + self.assertEqual(result["modified"], []) + + def test_apply_prunes_empty_extra_directory_ancestors(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + source_root = root / "source" + repository = root / "repo" + target_root = repository / "owned" + source_root.mkdir() + (target_root / "extra/empty").mkdir(parents=True) + source = carry.inventory(source_root, ["**/*"]) + result = carry.compare(source, carry.inventory(target_root, ["**/*"])) + + changes = carry.apply_tree(source, target_root, repository, result) + + self.assertEqual(changes, ["remove owned/extra/empty", "remove owned/extra"]) + + def test_apply_prunes_directories_emptied_by_extra_file_removal(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + source_root = root / "source" + repository = root / "repo" + target_root = repository / "owned" + source_root.mkdir() + (target_root / "extra").mkdir(parents=True) + (target_root / "extra/file.txt").write_text("extra", encoding="utf-8") + source = carry.inventory(source_root, ["**/*"]) + result = carry.compare(source, carry.inventory(target_root, ["**/*"])) + + changes = carry.apply_tree(source, target_root, repository, result) + final = carry.compare(source, carry.inventory(target_root, ["**/*"])) + + self.assertEqual(changes, ["remove owned/extra/file.txt", "remove owned/extra"]) + self.assertEqual(final["extra"], []) + self.assertEqual(final["extraDirectories"], []) + + +class CarryManifestTests(unittest.TestCase): + def test_selector_excludes_inapplicable_declaration(self) -> None: + self.assertFalse(carry.applicable(["python"], {"csharp", "release"})) + + def test_rejects_overlapping_targets(self) -> None: + declarations = [ + { + "source": "source-a", + "target": ".github", + "fidelity": "verbatim-tree", + "appliesTo": "*", + "include": ["**/*"], + "prune": False, + }, + { + "source": "source-b", + "target": ".github/skills", + "fidelity": "verbatim-tree", + "appliesTo": "*", + "include": ["**/*"], + "prune": False, + }, + ] + + with ( + tempfile.TemporaryDirectory() as temp, + self.assertRaisesRegex(carry.CarryError, "overlapping tree declarations"), + ): + carry.validate_declarations(declarations, pathlib.Path(temp)) + + def test_rejects_malformed_tree_declarations(self) -> None: + valid = { + "source": "source", + "target": "target", + "fidelity": "verbatim-tree", + "appliesTo": "*", + "include": ["**/*"], + "prune": False, + } + malformed = [ + None, + {key: value for key, value in valid.items() if key != "source"}, + {**valid, "unknown": True}, + {**valid, "include": []}, + {**valid, "prune": "false"}, + ] + with tempfile.TemporaryDirectory() as temp: + for declaration in malformed: + with self.subTest(declaration=declaration), self.assertRaises(carry.CarryError): + carry.validate_declarations([declaration], pathlib.Path(temp)) + + def test_rejects_target_outside_repository(self) -> None: + with ( + tempfile.TemporaryDirectory() as temp, + self.assertRaisesRegex(carry.CarryError, "repository-relative"), + ): + carry.relative_root(pathlib.Path(temp), "../outside") + + def test_rejects_normalized_parent_segment(self) -> None: + with ( + tempfile.TemporaryDirectory() as temp, + self.assertRaisesRegex(carry.CarryError, "repository-relative"), + ): + carry.relative_root(pathlib.Path(temp), "inside/../target") + + def test_rejects_repository_root(self) -> None: + with ( + tempfile.TemporaryDirectory() as temp, + self.assertRaisesRegex(carry.CarryError, "below the repository root"), + ): + carry.relative_root(pathlib.Path(temp), ".") + + def test_rejects_symlinked_declared_root(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + real = root / "real" + real.mkdir() + (root / "linked").symlink_to(real, target_is_directory=True) + with self.assertRaisesRegex(carry.CarryError, "symlink is not allowed"): + carry.relative_root(root, "linked/tree") + + def test_target_identity_and_unrelated_dirty_state(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = pathlib.Path(temp) + remote = root / "remote.git" + clone = root / "clone" + worktree = root / "worktree" + subprocess.run(["git", "init", "--bare", remote], check=True, capture_output=True) + subprocess.run(["git", "clone", remote, clone], check=True, capture_output=True) + for key, value in (("user.name", "Test"), ("user.email", "test@example.invalid")): + subprocess.run(["git", "-C", clone, "config", key, value], check=True) + (clone / "seed.txt").write_text("seed", encoding="utf-8") + subprocess.run(["git", "-C", clone, "add", "seed.txt"], check=True) + subprocess.run( + ["git", "-C", clone, "commit", "-m", "Seed"], check=True, capture_output=True + ) + subprocess.run(["git", "-C", clone, "branch", "-M", "develop"], check=True) + subprocess.run( + ["git", "-C", clone, "push", "-u", "origin", "develop"], + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "-C", clone, "worktree", "add", "-b", "feature/test", worktree, "develop"], + check=True, + capture_output=True, + ) + owned = worktree / "owned" + + with self.assertRaisesRegex(carry.CarryError, "origin does not match"): + carry.verify_target(worktree, {"url": str(root / "other.git")}, [owned]) + + (worktree / "unrelated.txt").write_text("dirty", encoding="utf-8") + with self.assertRaisesRegex(carry.CarryError, "unrelated changes"): + carry.verify_target(worktree, {"url": str(remote)}, [owned]) + (worktree / "unrelated.txt").unlink() + + owned.mkdir() + subprocess.run(["git", "-C", worktree, "mv", "seed.txt", "owned/seed.txt"], check=True) + with self.assertRaisesRegex(carry.CarryError, "unrelated changes"): + carry.verify_target(worktree, {"url": str(remote)}, [owned]) + + +if __name__ == "__main__": + unittest.main() diff --git a/spec/audit.py b/spec/audit.py index 76e1d9ef..3beedeb0 100755 --- a/spec/audit.py +++ b/spec/audit.py @@ -25,6 +25,7 @@ import argparse import base64 +import fnmatch import functools import hashlib import itertools @@ -103,7 +104,17 @@ def hub_only_paths(spec): So only a `retire` disposition in spec/divergences.json asserts a deletion, and an untriaged hit asks for the file to be read. """ - return hub_tracked() - {e["path"] for e in spec["files"]["baseline"]} + declared = {e["path"] for e in spec["files"]["baseline"]} + tree_paths = set() + for declaration in spec["files"].get("trees", []): + root = declaration["source"].rstrip("/") + "/" + tree_paths.update( + path + for path in hub_tracked() + if path.startswith(root) + and tree_path_included(path.removeprefix(root), declaration["include"]) + ) + return hub_tracked() - declared - tree_paths def gap_dispositions(spec): @@ -123,8 +134,8 @@ def gap_dispositions(spec): return out -def repo_tree(slug, ground_head): - """Every blob path on a repo's ground-truth branch, or None where it could not be read in full. +def repo_tree_entries(slug, ground_head): + """Every blob path and object SHA on a repo branch, or None when it cannot be read in full. The trees endpoint takes a tree sha rather than a ref name, so the branch payload already read is resolved to its tree instead of passing the branch name, which 404s. @@ -137,7 +148,31 @@ def repo_tree(slug, ground_head): tree = gh(f"repos/{slug}/git/trees/{sha}?recursive=1", ok404=True) if not tree or "tree" not in tree or tree.get("truncated"): return None - return {n["path"] for n in tree["tree"] if n.get("type") == "blob"} + return {n["path"]: n.get("sha") for n in tree["tree"] if n.get("type") == "blob"} + + +def repo_tree(slug, ground_head): + entries = repo_tree_entries(slug, ground_head) + return None if entries is None else set(entries) + + +def git_blob_sha(content): + header = f"blob {len(content)}\0".encode() + return hashlib.sha1(header + content).hexdigest() + + +@functools.cache +def canonical_blob_sha(path): + return git_blob_sha((ROOT / path).read_bytes()) + + +def tree_path_included(path, patterns): + return any( + pattern == "**/*" + or fnmatch.fnmatchcase(path, pattern) + or pathlib.PurePosixPath(path).match(pattern) + for pattern in patterns + ) def hub_name(): @@ -1329,6 +1364,19 @@ def git_file_history(rel_path): return out +@functools.cache +def git_blob_in_file_history(rel_path, blob_sha): + """Whether a blob occurred in a path's hub history.""" + result = subprocess.run( + ["git", "log", "--format=%H", f"--find-object={blob_sha}", "--", rel_path], + cwd=ROOT, + capture_output=True, + text=True, + check=False, + ) + return result.returncode == 0 and bool(result.stdout.strip()) + + @functools.cache def hub_last_change(rel_path): """The hub checkout's last commit touching rel_path, as (iso_date, short_sha), or None if untracked. @@ -1860,12 +1908,87 @@ def audit_repo(entry, spec, branch=None): ) ) + # --- Manifest-owned verbatim trees --- + carried_entries = repo_tree_entries(slug, ground_head) + if carried_entries is None: + findings.append( + ( + "DRIFT", + f"tree: could not read the file tree for {slug}@{ground} in full, so verbatim trees are undecided", + ) + ) + elif entry.get("name") != HUB_NAME: + for declaration in spec["files"].get("trees", []): + if not applies(declaration.get("appliesTo", "*"), sel): + continue + source_root = declaration["source"].rstrip("/") + target_root = declaration["target"].rstrip("/") + source_files = { + path.removeprefix(source_root + "/"): path + for path in hub_tracked() + if path.startswith(source_root + "/") + and tree_path_included(path.removeprefix(source_root + "/"), declaration["include"]) + } + target_files = { + path.removeprefix(target_root + "/"): path + for path in carried_entries + if path.startswith(target_root + "/") + } + for relative in sorted(set(source_files) - set(target_files)): + findings.append(("LETTER", f"tree: {target_root}/{relative} is absent on {ground}")) + for relative in sorted(set(source_files) & set(target_files)): + source_path = source_files[relative] + try: + source_sha = canonical_blob_sha(source_path) + except OSError as exc: + findings.append( + ( + "DRIFT", + f"tree: canonical {source_path} is unreadable, comparison undecided: {exc}", + ) + ) + continue + target_path = target_files[relative] + target_sha = carried_entries[target_path] + if source_sha == target_sha: + continue + if not target_sha: + findings.append( + ( + "DRIFT", + f"tree: {target_path} differs but its blob identity is unreadable, comparison undecided", + ) + ) + continue + verdict = ( + "stale" if git_blob_in_file_history(source_path, target_sha) else "modified" + ) + if verdict == "stale": + findings.append( + ( + "DRIFT", + f"tree: {target_path} matches a past hub revision, not the current canonical", + ) + ) + else: + findings.append( + ( + "DRIFT", + f"tree: {target_path} differs from the canonical and matches no past hub revision", + ) + ) + if declaration.get("prune"): + for relative in sorted(set(target_files) - set(source_files)): + findings.append( + ("DRIFT", f"tree: {target_root}/{relative} is extra under a pruned target") + ) + # --- Hub-only files a repo carries and should not --- # The manifest declares what a repo carries, so a path the hub tracks and the manifest omits is hub-hosted content per GOVERNANCE.md "Hub-Hosted Tooling". # A downstream copy of one is drift whose remedy is a deletion rather than a re-vendor, which no other check reports: every check above reads a path the manifest names, so nothing looks at what a repo carries beyond the baseline. # Skipped for the hub, whose own tracked files are the source and are all "hub-only" by construction. if entry.get("name") != HUB_NAME: - carried = repo_tree(slug, ground_head) + carried = None if carried_entries is None else set(carried_entries) if carried is None: findings.append( ( diff --git a/spec/fidelity-model.md b/spec/fidelity-model.md index 2560f61c..a921cafc 100644 --- a/spec/fidelity-model.md +++ b/spec/fidelity-model.md @@ -6,7 +6,7 @@ How faithfully each carried unit must survive the carry, and how that is verifie Carried content is a class with virtual functions. The **fixed** part is the interface: when a thing is invoked, what it is named, and where it is wired. The **overridable** part is the implementation body, which a repo replaces to fit its own targets. Validation must allow the override while detecting a change to the interface or to content meant to stay fixed. Integrity is by **content hash, never a version number**. A version stamp is a claim a repo can keep while editing the body, so it is never trusted for detection. -## The Four Fidelity Levels +## The Fidelity Levels Each [`spec/files.json`][files] entry declares one `fidelity`, defaulting to `presence`. @@ -15,6 +15,8 @@ Each [`spec/files.json`][files] entry declares one `fidelity`, defaulting to `pr - **verbatim** - byte-identical to the hub's canonical after line-ending, action-pin, and job-needs normalization. The audit content-hashes the downstream copy against canonical. It applies to a whole file, a workflow job region (a job selected by key), or a Markdown section region (a `## heading` block selected by name). The section granularity lets one file be **intent overall while a few of its sections are verbatim**. A universal rule block stays byte-identical fleet-wide even though the rest of the document is a repo-adapted paraphrase, so a stale section or a missing rule is caught while its heading still passes the presence check. - **interface** - an overridable body that must honor a named contract. The audit checks the contract by name and wiring, never the body. +A declaration in `trees[]` uses **verbatim-tree**. Each included source file exists at the corresponding target path with identical bytes. A pruned target contains no extra path. The tree root exists even when the included source inventory is empty. Tree target roots never overlap because two declarations cannot own the same path safely, regardless of whether either declaration prunes extras. + Fidelity is a declared field defaulting to `presence`, never inferred from `whole`/`placeholders`. `.editorconfig` and `.markdownlint-cli2.jsonc` are both whole with no placeholders yet sit at opposite fidelity, because the discriminator is governance, not field shape. ## Why Each Unit Sits Where It Does @@ -34,6 +36,8 @@ Once a workflow's job graph moves into the hub, the carried unit is a caller stu A verbatim check compares content by hash after **line-ending, action-pin, and job-needs normalization**. EOL variance is governed by the line-ending rules, a `uses: @` pin (with its trailing `# vN` comment) is Dependabot-owned and bumped per repo, and a job's `needs:` list is pruned per repo to its vendored targets (an unvendored name fails the workflow to load), so all three are governed drift rather than a fidelity deviation. This keeps a verbatim workflow job region (the `github-release` job) from flagging on a routine action bump or a legitimate needs-prune while still catching a real structural fork. It does **not** mask placeholders: a verbatim unit carries none. The files that declare a `placeholders` list (for example `.github/copilot-instructions.md` with ``, ``, ``) are fidelity `intent`, judged by hand and never hashed. Masking could not serve a hash anyway. A downstream copy holds the substituted value (`ptr727`), not the token (``), so masking the token in the canonical alone would guarantee a mismatch. A verbatim unit that ever needed a per-repo substitution would require template-matching (the canonical as a pattern, the copy as an instance), not this content hash. None does today. +`verbatim-tree` applies no normalization. It compares file bytes because a fully owned generated tree has no per-repository substitutions or Dependabot-owned regions. + ## Stale Versus Modified A verbatim mismatch is one of two things, told apart **by hash, not by a version**. The audit hashes each past revision of the hub's canonical from its own git history. If the downstream copy matches a **past** canonical revision, the base advanced and the copy is **stale**, so re-vendor it. If it matches **no** revision the base ever produced, the repo **modified fixed content**, so review it. A version stamp could claim to be current while being neither, so it is demoted to a human-facing label and never consulted for integrity. diff --git a/spec/files.json b/spec/files.json index c873e82f..2d084081 100644 --- a/spec/files.json +++ b/spec/files.json @@ -1,6 +1,9 @@ { "$schema": "./files.schema.json", - "note": "The standardization baseline: files and sections a fleet repo is expected to carry, and their intent authority. The audit mechanically checks presence (letter). Equivalence (intent) is judged by hand, and a section for an absent language or target is N/A. Each entry, and each section, carries an appliesTo selector - see spec/scope-model.md for the scope model and selector vocabulary. Each entry also has a fidelity (presence by default, or intent, verbatim, interface) governing how faithfully the content is checked - see spec/fidelity-model.md. The per-section fidelity of AGENTS.md and GOVERNANCE.md, which sections are verbatim fleet-law and which are repo-specific, is defined in spec/section-model.md.", + "note": "The standardization baseline: files, sections, and fully owned trees a fleet repo is expected to carry. See spec/scope-model.md for applicability and spec/fidelity-model.md for fidelity.", + "trees": [ + { "source": ".github/skills", "target": ".github/skills", "fidelity": "verbatim-tree", "appliesTo": "*", "include": ["**/*"], "prune": true } + ], "baseline": [ { "path": "AGENTS.md", "fidelity": "intent", "sections": [{ "name": "Fleet Bootstrap", "fidelity": "verbatim" }, { "name": "Context and Delegation Discipline", "fidelity": "verbatim" }, { "name": "Where the Rules Live", "fidelity": "verbatim" }], "intentRef": "AGENTS.md", "appliesTo": "*" }, diff --git a/spec/files.schema.json b/spec/files.schema.json index 5786890c..11602265 100644 --- a/spec/files.schema.json +++ b/spec/files.schema.json @@ -3,10 +3,27 @@ "$id": "https://github.com/ptr727/ProjectTemplate/spec/files.schema.json", "title": "Standardization baseline files", "type": "object", - "required": ["baseline"], + "required": ["baseline", "trees"], "properties": { "$schema": { "type": "string" }, "note": { "type": "string" }, + "trees": { + "type": "array", + "items": { + "type": "object", + "required": ["source", "target", "fidelity", "appliesTo", "include", "prune"], + "additionalProperties": false, + "properties": { + "source": { "type": "string", "minLength": 1, "not": { "const": "." } }, + "target": { "type": "string", "minLength": 1, "not": { "const": "." } }, + "fidelity": { "const": "verbatim-tree" }, + "appliesTo": { "type": ["string", "array"], "items": { "type": "string" }, "minItems": 1 }, + "include": { "type": "array", "items": { "type": "string", "minLength": 1 }, "minItems": 1 }, + "prune": { "type": "boolean" }, + "allowHubTarget": { "type": "boolean" } + } + } + }, "baseline": { "type": "array", "items": { diff --git a/spec/validate.py b/spec/validate.py index 55265f2f..3d151e6c 100755 --- a/spec/validate.py +++ b/spec/validate.py @@ -7,6 +7,7 @@ dry-run the CI lint job runs; it needs no third-party packages. """ +import fnmatch import json import pathlib import re @@ -471,6 +472,97 @@ def check_selector(where, applies_to): if not isinstance(baseline, list): errors.append("files.json: 'baseline' must be an array") baseline = [] + + if "trees" not in files: + errors.append("files.json: missing required 'trees' array") + trees = files.get("trees", []) + if not isinstance(trees, list): + errors.append("files.json: 'trees' must be an array") + trees = [] + validated_trees = [] + for tree in trees: + if not isinstance(tree, dict): + errors.append(f"files.json: tree declaration {tree!r} is not an object") + continue + source = tree.get("source") + target = tree.get("target") + if not isinstance(source, str) or not source: + errors.append(f"files.json: tree declaration has an invalid source: {tree!r}") + continue + if not isinstance(target, str) or not target: + errors.append(f"files.json: tree declaration has an invalid target: {tree!r}") + continue + for field, value in (("source", source), ("target", target)): + parts = pathlib.PurePosixPath(value).parts + if value == "." or value.startswith("/") or ".." in parts: + errors.append( + f"files.json: tree {source} {field} '{value}' must be below the repository root (no leading /, . or ..)" + ) + if tree.get("fidelity") != "verbatim-tree": + errors.append(f"files.json: tree {source} fidelity must be 'verbatim-tree'") + if "appliesTo" not in tree: + errors.append(f"files.json: tree {source} is missing required appliesTo") + else: + check_selector(f"tree {source}", tree["appliesTo"]) + include = tree.get("include") + if not is_str_list(include) or not include: + errors.append(f"files.json: tree {source} include must be a non-empty array of strings") + if not isinstance(tree.get("prune"), bool): + errors.append(f"files.json: tree {source} prune must be a boolean") + if "allowHubTarget" in tree and not isinstance(tree["allowHubTarget"], bool): + errors.append(f"files.json: tree {source} allowHubTarget must be a boolean") + if not (ROOT / source).is_dir(): + errors.append( + f"files.json: tree canonical source {source} is missing or not a directory" + ) + validated_trees.append(tree) + + for index, left in enumerate(validated_trees): + left_target = pathlib.PurePosixPath(left["target"]) + for right in validated_trees[index + 1 :]: + right_target = pathlib.PurePosixPath(right["target"]) + overlaps = ( + left_target == right_target + or left_target in right_target.parents + or right_target in left_target.parents + ) + if overlaps: + errors.append( + f"files.json: tree targets {left_target} and {right_target} have overlapping ownership" + ) + + copilot_path = ROOT / ".github/copilot-instructions.md" + try: + copilot_instructions = copilot_path.read_text(encoding="utf-8") + except OSError as exc: + errors.append(f"files.json: cannot read {copilot_path.relative_to(ROOT)}: {exc}") + copilot_instructions = "" + named_skills = set( + re.findall(r"\.github/skills/[A-Za-z0-9_-]+/SKILL\.md", copilot_instructions) + ) + for path in sorted(named_skills): + if not (ROOT / path).is_file(): + errors.append(f"files.json: Copilot instructions reference missing skill path {path}") + skill_path = pathlib.PurePosixPath(path) + carried = False + for tree in validated_trees: + target_root = pathlib.PurePosixPath(tree["target"]) + include = tree.get("include") + if target_root not in skill_path.parents or not is_str_list(include): + continue + relative = skill_path.relative_to(target_root).as_posix() + carried = any( + pattern == "**/*" + or fnmatch.fnmatchcase(relative, pattern) + or pathlib.PurePosixPath(relative).match(pattern) + for pattern in include + ) + if carried: + break + if not carried: + errors.append( + f"files.json: Copilot instructions reference skill path {path} outside every carried tree include" + ) for item in baseline: if not isinstance(item, dict): errors.append(f"files.json: baseline entry {item!r} is not an object")