Add fidelity-honesty analysis tool (owner-run) by ptr727 · Pull Request #387 · ptr727/ProjectTemplate

ptr727 · 2026-07-22T13:46:57Z

A read-only fleet analysis that checks the manifest's declared fidelities against reality — the declared→verified leap one level up from the verbatim engine (which verifies content). Reuses spec/audit.py's machinery. Owner-run, not wired into CI.

What it reports

intent units byte-uniform fleet-wide → candidates to promote to verbatim (gaining free drift-detection).
verbatim units with non-stale downstream divergence → mis-label or real drift.
Manifest gaps — files carried by a reference adopter but absent from the manifest (the configure.sh bug class).

First-run value (findings, not fixed here)

3 ruleset payloads (repo-config/{develop,operational/develop,main}.json) are uniform → verbatim candidates.
.markdownlint-cli2.jsonc and configure.sh carry real downstream drift (a re-vendor worklist).
.editorconfig-checker.json is carried but untracked — a new manifest gap.

Follow-ups (each its own decision) are tracked separately. This PR just lands the tool.

🤖 Generated with Claude Code

Read-only fleet analysis that checks the manifest's declared fidelities against reality - the declared-to-verified leap one level up from the verbatim engine (which verifies content). Reuses spec/audit.py's machinery. For each intent/verbatim unit it fetches every applicable cataloged repo's copy and reports: intent units that are byte-uniform fleet-wide (candidates to promote to verbatim, gaining free drift-detection), verbatim units with non-stale downstream divergence (mis-label or drift), and a manifest-gap pass (files carried by a reference adopter but absent from the manifest - the configure.sh bug class). Not wired into CI; prototype for review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Adds an owner-run, read-only fleet analysis script that reuses spec/audit.py helpers to compare spec/files.json declared fidelities (intent/verbatim) against observed downstream content hashes, and to detect manifest gaps via a reference adopter.

Changes:

Add spec/fidelity_honesty.py to compute per-unit fleet spread, promotion candidates, and potential mislabels.
Add a manifest-gap pass that compares hub files against a reference adopter's repo tree.
Print a human-readable report summarizing the findings.

- manifest_gap_pass resolved the git/trees endpoint with a branch name; that can 404 and silently drop the check. Resolve the branch to its tree SHA first, as audit.py does (also removes a possibly-unbound local). - Pin spec/fidelity_honesty.py to LF in .editorconfig alongside its audit.py / validate.py siblings, so its shebang stays executable. - Reword "byte-uniform" -> "content-identical (after EOL normalization)": the comparison is content_hash, which normalizes line endings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

…ributes (#387) - The manifest-gap pass excluded paths with `.startswith(".git")`, which also swallowed `.github/`, `.gitattributes`, and `.gitignore` - hiding real gaps. And the hub-side `.git/` substring exclusion was path-separator-dependent. Exclude the .git metadata dir by path component (`".git" not in p.parts`) and drop the startswith filter. This immediately surfaced two previously-hidden gaps: .github/workflows/publish-release.yml and validate-task.yml are carried but untracked. - Add spec/fidelity_honesty.py to the .gitattributes LF-pin, matching its audit.py / validate.py / install.py siblings (git-enforced, not just editor). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

…le" bucket (#387) - The verbatim-candidate detector treated any present copy as evidence of uniformity, so a unit that was entirely stale/unavailable could be flagged. Require at least one confirmed match with the current canonical (still allowing stale copies, which verbatim would flag "stale -> re-vendor"), and reword the header to stop implying "fleet-wide uniform". - Rename the "absent" bucket to "unavailable": fetch() returns None for both a missing file and a present-but-non-inline response (too large / encoding "none"), so "absent" over-claimed. The report now labels it "absent or non-inline". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

spec/fidelity_honesty.py:55

The inline comment for promote still says these intent units are "uniform fleet-wide", but the actual criteria only requires >=1 confirmed match with the current canonical and 0 "differs" (stale/unavailable are allowed). This makes the comment misleading for future readers.

 spreads = [] # (entry, spread dict) for the full table
promote = [] # intent units that are uniform fleet-wide
mislabel = [] # verbatim units that diverge non-stale

…lk (#387) manifest_gap_pass built the hub-file set with ROOT.rglob('*'), a filesystem walk that pulls in untracked local cruft (__pycache__ from running the tool, a local .venv) and makes the gap report depend on working-tree state. Use git ls-files - the tracked set, which is what "carried" actually means and is deterministic regardless of local files. Same gaps result, without the walk's noise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

spec/fidelity_honesty.py:106

manifest_gap_pass() returns an empty gaps list when the branch/tree lookup fails (404 / missing fields). main() then prints "none - the manifest covers...", which is a false negative because the check did not actually run. Also, the git/trees API can return {"truncated": true}; in that case the gap set would be incomplete and should fail loud rather than reporting "none".

 br = audit.gh(f"repos/{slug}/branches/{ground}", ok404=True)
if not br or "commit" not in br:
return slug, []
tree = audit.gh(f"repos/{slug}/git/trees/{br['commit']['commit']['tree']['sha']}?recursive=1", ok404=True)
if not tree or "tree" not in tree:
return slug, []

spec/fidelity_honesty.py:54

The inline comment for promote says "uniform fleet-wide", but the logic intentionally allows stale/unavailable repos and only requires >=1 canonical match and 0 hand-modified copies. Updating the comment avoids misreading what promote represents.

 promote = [] # intent units that are uniform fleet-wide

An empty hub-file set would make the manifest-gap report show "none" - a false clean. Raise on a non-zero git ls-files exit instead, matching audit.py's gh(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

spec/fidelity_honesty.py:54

The inline comment says intent units in promote are "uniform fleet-wide", but the promote condition allows stale/unavailable copies (it only requires >=1 match and 0 confirmed divergent copies). This comment is misleading and contradicts the logic below.

 promote = [] # intent units that are uniform fleet-wide

…#387) If the reference adopter's branch or tree lookup returned nothing, manifest_gap_pass returned an empty gaps list, which main() reports as "none" - a false clean. Raise instead, matching the git ls-files guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

spec/fidelity_honesty.py:55

The promote list no longer implies fleet-wide uniformity (units can be unavailable in some repos), but the comment still says "uniform fleet-wide", which is misleading.

 promote = [] # intent units that are uniform fleet-wide
mislabel = [] # verbatim units that diverge non-stale

…ons (#387) Three semicolons in this file's comments and output string joined independent clauses, which the AGENTS.md prose rule bans. Recast to commas/periods. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot