Add Regression Test Suite and Reduced-Corpus Tooling - #855
Merged
Conversation
Add the ZFS-clone regression harness and the standalone Python tooling that verifies PlexCleaner's processing decisions stay consistent across versions against a curated media collection: catalog generation, reduced-corpus build with a prove-equivalence gate, decode-signature location, and physical-shape coverage audit. No media or media filenames are committed. Media-specific reduction rules are externalized to a JSON file that lives with the media and is read and written by the tooling; the repo ships only a synthetic example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ruff and mypy configuration for the RegressionTests Python utilities (mirroring the audited Financial-Modeling conventions), a CI lint step in the validate gate that runs them via uvx at pinned versions, matching VSCode lint tasks for local parity, and gitignore entries for the Python caches. This is the first Python in the repo; the tooling is stdlib-only and Python source follows the repo default CRLF line ending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the harness usage, the full-file-scan rationale, the tooling, the external rules format and generate-on-demand flow, the cutter ladder and surgical IETF rungs, naming conventions, the update-validate-snapshot loop, and the physical-shape coverage audit. All media titles are generalized. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the superseded ad-hoc regression section in the README with a pointer to RegressionTests/, add the directory and a Python line-ending note to AGENTS.md, add a Testing pointer in ARCHITECTURE.md, and record the addition in HISTORY.md under the unreleased 3.21 version. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the ruff, mypy, Pylance, and Python extension recommendations so the editor surfaces the RegressionTests lint and type-check tooling. Recommendations only; no Python workspace settings are needed for the stdlib-only scripts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a standalone RegressionTests/ suite to preserve a reproducible, cross-version regression-corpus workflow for PlexCleaner, including a ZFS-clone harness plus stdlib-only Python tooling to catalog issues, build/validate a reduced corpus, locate decode signatures, and audit physical error-shape coverage. It also wires Python lint/typecheck into CI and VS Code tasks, and updates top-level docs to point to the new suite.
Changes:
- Add
RegressionTests/harness (RegressionTest.sh) and Python utilities (catalog_corpus.py,reduce_corpus.py,locate_issue.py,audit_physical.py,corpus_common.py) plus documentation and a synthetic reduction-rules example. - Add Python lint/typecheck config (
pyproject.toml), CI validate step (uvx + pinned ruff/mypy), and VS Code tasks. - Update repo docs (
README.md,ARCHITECTURE.md,AGENTS.md,HISTORY.md) to document and link the regression suite; add Python cache ignores.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| RegressionTests/RegressionTest.sh | ZFS-clone + Docker-image regression harness; versioned results/log output; optional plugin build/run. |
| RegressionTests/reduction-rules.example.json | Synthetic example schema for external, media-adjacent reduction rules. |
| RegressionTests/reduce_corpus.py | Reduced-corpus generator/validator with equivalence gate (state/detections/signatures) and cutter ladder. |
| RegressionTests/README.md | Full documentation for regression suite layout, harness usage, and tooling workflows. |
| RegressionTests/pyproject.toml | Ruff + mypy configuration for the new Python tooling. |
| RegressionTests/locate_issue.py | Decode-signature localization tooling; optionally writes region windows into external rules file. |
| RegressionTests/corpus_common.py | Shared parsing + classification logic for logs/signatures and clip/surgery helpers. |
| RegressionTests/catalog_corpus.py | Generates catalog.json from a versioned regression run (ground-truth issue catalog). |
| RegressionTests/audit_physical.py | Audits physical ffmpeg error-shape coverage for reduced clips and annotates catalog. |
| README.md | Replaces the long manual regression-testing recipe with links to RegressionTests/. |
| HISTORY.md | Notes addition of RegressionTests suite/tooling (no application behavior change). |
| ARCHITECTURE.md | Documents regression testing capability and links to the suite README. |
| AGENTS.md | Documents CRLF conventions for Python/TOML and adds RegressionTests to project structure overview. |
| .vscode/tasks.json | Adds Ruff/Ruff-format/Mypy tasks and includes them in “Lint: All (CI parity)”. |
| .gitignore | Ignores Python caches for RegressionTests tooling. |
| .github/workflows/validate-task.yml | Adds uvx-based ruff/format/mypy validation for RegressionTests in CI. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Address Copilot review feedback: - reduce_corpus.py: process_clip now returns a None State map when a run does not complete (missing or unreadable results file) and the caller treats that rung as failed, so a failed run can no longer masquerade as an empty-State PASS and weaken the equivalence gate. - RegressionTest.sh: drop sudo from the zfs calls; the script already exits unless run as uid 0, so sudo was redundant and would break on a root host without sudo installed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address Copilot review feedback: the utilities are CRLF (the audited repo convention) and are invoked via python3 or uvx, never executed directly, and they carry no executable bit. A shebang on a CRLF file is non-functional and misleading (direct execution would fail on the CR), so remove it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Address Copilot review feedback: - RegressionTest.sh: the version results directory is populated as root but the containers run as nobody:users and write logs and results into it; chown it to that user before the container runs so the writes succeed on any host, independent of the parent directory's ownership or ACLs. - pyproject.toml and README.md: pin the ruff and mypy versions in the example commands to match CI and the VSCode tasks, so local runs cannot diverge from CI on an unpinned newer tool. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Address Copilot review feedback: - RegressionTest.sh: roll the test clone back to its pristine @backup via an EXIT trap so an aborted process or plugin run under set -e never leaves the shared dataset mutated; the trap is disarmed after the clean restore. Also complete the mode-validation usage string with the corpus and plugin args. - locate_issue.py: write_region now sets the top-level schema field so a generated rules file matches the shipped reduction-rules.example.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep the ruff and mypy version pins in the CI workflow (where they can be bumped) and run the latest tools in the VSCode tasks and the local-run examples in the README and pyproject comment. Pinning outside CI would drift silently with no bump path, which is worse than a small, visible local-versus-CI gap; a local linter that never falls behind is the safer default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
The comment claimed the CI pins match the editor tasks, but the VSCode tasks now run the latest tools; only CI pins exact versions. Reword to state the intentional local-versus-CI gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Address Copilot review feedback: process_clip captured the container stdout and stderr to DEVNULL, so a failed run (a crash or a container that never starts) gave no diagnostic. Capture the combined output and print its tail with the exit code when no results file is produced, or the parse error when the results file is unreadable, so a failed reduction rung is debuggable. Output is discarded on success. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Jul 17, 2026
Closed
ptr727 added a commit
to ptr727/ProjectTemplate
that referenced
this pull request
Jul 17, 2026
Fixes#339. Registers the fleet's first mixed-language shape (PlexCleaner: a .NET console app plus a stdlib-only Python tooling subtree) and codifies how the `csharp` and `python` baselines coexist. Reference implementation: ptr727/PlexCleaner#855. Per the issue, this is **not** a new `polyglot` type - a mixed repo is expressed as `types: ["csharp","console","docker","python"]` and the audit runs the union of the per-type checks. The work is: register the shape, codify the coexistence rules, add a matrix row. ## Three decisions (settled with the owner before writing) 1. **Two python profiles, one type, split by a third-party runtime import.** PROJECT = a PEP 621 uv project with runtime deps + committed LF-pinned `uv.lock` (the existing Financial-Modeling shape). SCRIPTS = stdlib-only utility scripts embedded in a non-Python repo, run via `uvx`, no lockfile, `pyproject.toml` carrying only `[tool.ruff]`/`[tool.mypy]`. Detected structurally from `pyproject.toml`. 2. **Tool-version pinning for SCRIPTS is CI-only.** CI pins exact versions in the `uvx` command (`uvx ruff@X`, bumpable there); tasks/README run latest - a deliberate CI-vs-local gap so local never silently falls behind. PROJECT keeps pinning through `uv.lock` + `uv sync --frozen` (unchanged). 3. **Coverage N/A for SCRIPTS.** `python.coverage.codecov` is N/A for a lint/type-only subtree (no pytest); `codecov.yml` presence stays required by any co-present type with tests (the C# side here). ## Changes | File | Change | |---|---| | `registry/repos.json` | PlexCleaner `types` += `python`; driftNote records the SCRIPTS subtree + which checks go N/A | | `spec/project-types.json` | `python` type: `profileNote` + new `python.profile.detect`; SCRIPTS-profile N/A wording on `pyright`/`coverage`/`uvlock`; new `python.scripts.uvx` check | | `CODESTYLE.md` | "Two profiles" paragraph (project vs scripts: uvx, tool-config-only pyproject, mypy, CI-pinned/local-latest, `.py` on repo EOL default, coverage N/A) | | `README.md` `## Rules` | Python bullet names the type-checker choice; new "If Both C# and Python" both-apply subsection | | `reports/conformance-matrix.md` | `csharp` + `python` shape row (PlexCleaner) | | `cspell.json` | `bumpable`, `stdlib`, `uvx` | ## Issue items mapped - **(1) line endings** - CODESTYLE "Scripts" bullet states `.py`/`.toml` follow the repo CRLF default (only a shebang-executed script is LF-pinned), so nobody "fixes" `.py` to LF; `recurring.eol` already covers the general rule. - **(2) uvx-vs-uv-project** - `python.profile.detect` + CODESTYLE "Two profiles". - **(3) CI-only pinning** - `python.scripts.uvx` + CODESTYLE. - **(4) coverage** - `python.coverage.codecov` N/A wording. - **(5) tasks.json union** - confirmed, **no change needed**: `spec/files.json` already carries separate `csharp` and `python` `.vscode/tasks.json` references, so a mixed repo satisfies both by union (that is the intended audit behavior). - **(6) docs both-apply** - README "If Both C# and Python" + CODESTYLE "Two profiles". ## Validation - `python3 spec/validate.py` -> OK (21 cataloged, 0 backlog). - markdownlint + editorconfig-checker clean on changed files; CI-scoped cspell (README + HISTORY) clean. Related: #306 (closed - source-only Python adaptation gaps), #310 (open - the conformance-matrix row here is a `not-tested` entry pending that issue's cold-start self-test). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Preserve the reproducible regression-corpus process in the repo, as a standalone
suite under
RegressionTests/. It verifies that PlexCleaner's processingdecisions stay consistent across versions against a curated media collection.
RegressionTest.sh): provisions a disposable test dataset as azero-copy ZFS clone of the newest collection snapshot, processes it through one
Docker image tag, and writes per-version results for diffing.
catalog_corpus.pyderives a machine-readableissue catalog;
reduce_corpus.pybuilds a reduced collection behind aprove-equivalence gate (State + detections + signature class);
locate_issue.pyfinds where a decode signature lives and can record it into the rules file;
audit_physical.pyaudits physical-error-shape coverage;corpus_common.pyisthe shared library.
Copyright safety
No media or media filenames are committed. Media-specific reduction rules (cut
windows) are externalized to a JSON file that lives with the media and is read and
written by the tooling. The repo ships only
reduction-rules.example.jsonwithsynthetic placeholder names.
First Python in the repo
RegressionTests/pyproject.tomladds ruff + mypy config mirroring the auditedFinancial-Modeling conventions. CI runs them via
uvxat pinned versions in thevalidate gate, with matching VSCode lint tasks. Python source is CRLF (the repo
default; no
[*.py]override), and.gitignorecovers the Python caches. Afollow-up will file a ProjectTemplate issue to converge a canonical C#+Python type.
Version
Version stays 3.21 (unreleased); HISTORY.md records the addition.
Verification
ruff, ruff format, mypy, markdownlint (
**/*.md), cspell, and editorconfig-checkerall pass; a copyright gate over the committed tree finds no media titles.
🤖 Generated with Claude Code