diff --git a/.gitattributes b/.gitattributes index 09a8bf897..b03ee42f9 100644 --- a/.gitattributes +++ b/.gitattributes @@ -20,16 +20,37 @@ # apache-magpie--source.zip) honours `export-ignore`, so the # paths below are kept out of the signed source .zip that the [VOTE] # votes on. Keep this list to VCS/CI/editor metadata — never source. +# +# IMPORTANT — do NOT export-ignore a directory that shipped files +# still reference (by markdown link, `test -f`, or symlink). The +# committed agent-view symlink dirs (`.claude/skills/`, `.github/skills/`, +# `.kiro/skills/`) relay through `.agents/skills/*` → `skills/*`, and +# specs test those paths, so `.agents/` MUST ship or every relay dangles +# in the archive. Likewise `.github/ISSUE_TEMPLATE/` and +# `.github/PULL_REQUEST_TEMPLATE.md` are linked by skills, and +# `projects/_template/.gitignore` is example content — so only the true +# CI/dev entries under `.github/` are stripped, and `.gitignore` is +# anchored to the repo root. `release-verify-rc` re-checks the unpacked +# tarball (symlink-lint + validators + no-.pyc) so a regression here +# fails the RC before the [VOTE]. + +# Root-only VCS / dev-tool metadata (anchored so identically-named files +# deeper in the tree — e.g. projects/_template/.gitignore — still ship). +/.gitattributes export-ignore +/.gitignore export-ignore +/.pre-commit-config.yaml export-ignore +/.lychee.toml export-ignore +/.lycheecache export-ignore +/.markdownlint.json export-ignore +/.typos.toml export-ignore +/.zizmor.yml export-ignore +/.apache-magpie.session-state.json export-ignore -.gitattributes export-ignore -.gitignore export-ignore -.github/ export-ignore +# Editor metadata. .idea/ export-ignore -.agents/ export-ignore -.pre-commit-config.yaml export-ignore -.lychee.toml export-ignore -.lycheecache export-ignore -.markdownlint.json export-ignore -.typos.toml export-ignore -.zizmor.yml export-ignore -.apache-magpie.session-state.json export-ignore + +# CI / bot config only. The rest of `.github/` (ISSUE_TEMPLATE, +# PULL_REQUEST_TEMPLATE.md, and the skills relay) is referenced by +# shipped skills and resolves via `.agents/`, so it stays in the release. +.github/workflows/ export-ignore +.github/dependabot.yml export-ignore diff --git a/projects/magpie/release-build.md b/projects/magpie/release-build.md index 08a1420b0..9854aea2e 100644 --- a/projects/magpie/release-build.md +++ b/projects/magpie/release-build.md @@ -31,7 +31,11 @@ the only release artefact. ## Build invocation The canonical source artefact is a deterministic `git archive` of the -tagged tree — no VCS metadata, no build output: +tagged tree — no VCS metadata, no build output. **Never `zip -r` a +working directory**: that captures `__pycache__/*.pyc` and other test +cruft and produces a non-reproducible tarball (this was the rc1 `-1`). +`git archive` only ever includes tracked files from the tag, and honours +the `export-ignore` rules in [`.gitattributes`](../../.gitattributes): ```bash # From the release tag -rcN: diff --git a/skills/release-verify-rc/SKILL.md b/skills/release-verify-rc/SKILL.md index 64d59e8ae..a1914bbae 100644 --- a/skills/release-verify-rc/SKILL.md +++ b/skills/release-verify-rc/SKILL.md @@ -9,7 +9,9 @@ description: | Read-only pre-flight verification of a staged release candidate (RC) for ``. Checks artefact integrity (GPG signatures and checksums), Apache RAT licence headers, NOTICE/LICENSE completeness, - prohibited-binary absence, and version-string consistency. Emits a + prohibited-binary absence (including `.pyc` / `__pycache__`), + source-tree integrity (no dangling symlinks or broken internal + references), and version-string consistency. Emits a structured PASS / PASS-WITH-WARNINGS / FAIL report. Makes no state change; a `--post-to ` flag proposes a comment for explicit RM confirmation before any posting. @@ -431,9 +433,14 @@ binary-exclude list from `release-build.md`. Emit the paste-ready scan command: ```bash -# Example: find compiled Java class files and native shared libraries -find -type f \( -name "*.class" -o -name "*.jar" \ - -o -name "*.so" -o -name "*.dylib" -o -name "*.dll" -o -name "*.exe" \) +# Compiled Java class files, native shared libraries, AND compiled +# Python bytecode. `.pyc` / `__pycache__` must NEVER appear in a +# source release — their presence proves the artefact was zipped from +# a working tree that had run tests rather than exported clean from +# the tag (build via `git archive `, never `zip -r`). +find \( -type f \( -name "*.class" -o -name "*.jar" \ + -o -name "*.so" -o -name "*.dylib" -o -name "*.dll" -o -name "*.exe" \ + -o -name "*.pyc" \) -o -type d -name "__pycache__" \) -print ``` Emit the bare `find` with no `grep` post-filtering: the recipe must @@ -472,11 +479,78 @@ Return ONLY valid JSON with this structure: } ``` -`status` is `"FAIL"` if `prohibited_found` is non-empty. +`status` is `"FAIL"` if `prohibited_found` is non-empty. Any `.pyc` +file or `__pycache__` directory found is a hard `FAIL` (never an +`EXPECTED-BINARY`): it is both a prohibited binary and proof the +tarball was not exported clean from the tag. --- -## Step 7 — Version string consistency +## Step 7 — Source-tree integrity (dangling symlinks + broken references) + +The rc1 `-1` came from this class of defect, not from signatures or +RAT: committed agent-view symlinks (`.claude/skills/*`, `.kiro/skills/*`, +`.github/skills/*`) relayed into a directory (`.agents/`) that the +release stripped, so **every relay dangled**; and shipped files linked +to other stripped paths (`.github/` templates, `projects/_template/.gitignore`, +`.claude/skills/magpie-*/SKILL.md`), so the framework's own validators +failed. This step runs the same checks the framework applies to its own +tree, but **against the unpacked tarball**, so a packaging regression +(an `export-ignore` that strips a still-referenced path) fails the RC +before the `[VOTE]` rather than during it. + +Emit the paste-ready recipe (run from the unpacked dir; adapt tool +paths to the project — for Magpie the tools ship in-tree under +`tools/`): + +```bash +cd + +# 1. Dangling symlinks — every symlink must resolve inside the tarball. +find . -type l ! -exec test -e {} \; -print # any output = FAIL + +# 2. Internal reference / link integrity — run the project's own +# validators against the unpacked source (Magpie ships these): +uv run --project tools/symlink-lint symlink-lint . +uv run --project tools/skill-and-tool-validator skill-and-tool-validate +uv run --project tools/spec-validator spec-validate # if the project ships specs +``` + +Classify: + +- `PASS` — no dangling symlinks and every validator exits 0. +- `FAIL` — any dangling symlink, or any validator reports a broken + internal link / missing referenced file. This is a hard `FAIL`: + a release whose own files reference content that was stripped from + the artefact is incomplete. +- `SKIP` — the project ships no symlinks and no in-tree validators + (state this explicitly; do not silently pass). + +Do **not** post-filter the `find`; surface every dangling link so the +voter sees the full set. When a validator is not shippable in the +tarball, run it from a checkout of the *same tag* against the unpacked +dir instead, and note that in the report. + +Return ONLY valid JSON with this structure: + +```json +{ + "step": "source-tree-integrity", + "status": "PASS" | "FAIL" | "SKIP", + "dangling_symlinks": [""], + "validator_failures": [ + {"validator": "", "detail": ""} + ], + "paste_recipe": "" +} +``` + +`status` is `"FAIL"` if `dangling_symlinks` is non-empty or any +validator failed. + +--- + +## Step 8 — Version string consistency Read each file listed in `version_manifest_files` from `release-management-config.md` (e.g. `setup.cfg`, @@ -510,7 +584,7 @@ Return ONLY valid JSON with this structure: --- -## Step 8 — Hand-back verification report +## Step 9 — Hand-back verification report Aggregate the per-step results into a final report. @@ -604,7 +678,10 @@ supplied and the RM has not yet confirmed posting. | Step 5 FAIL — NOTICE or LICENSE absent | Source artefact build skipped packaging | RM fixes build process and cuts a new RC | | Step 5 WARN — material diff | Licence or attribution changed vs previous release | RM reviews diff; if intentional, document in planning issue | | Step 6 FAIL — prohibited binary | Binary sneaked into source artefact | RM removes binary, updates `.gitattributes` or build excludes, cuts new RC | -| Step 7 FAIL — version mismatch | Version bump missed one manifest file | RM fixes the manifest and cuts a new RC | +| Step 6 FAIL — `.pyc` / `__pycache__` present | Tarball zipped from a working tree that ran tests, not exported clean from the tag | RM rebuilds via `git archive ` (never `zip -r`), cuts new RC | +| Step 7 FAIL — dangling symlink | A committed symlink's target was stripped by `export-ignore` (or is otherwise absent) | RM fixes `.gitattributes` to ship the target (or drops the symlink), cuts new RC | +| Step 7 FAIL — broken internal reference | A shipped file links to a path stripped from the artefact | RM stops stripping the referenced path, or repoints the reference at shipped content, cuts new RC | +| Step 8 FAIL — version mismatch | Version bump missed one manifest file | RM fixes the manifest and cuts a new RC | ---