Fix ASVS 5.0 numbering across PRD skill, data index, and examples - #7
Merged
Conversation
The bundled data/asvs files are genuine ASVS 5.0 (V6 Authentication, V8 Authorization, V16 Logging), but the chapter index in data/asvs/README.md, the PRD skill's chapter mappings, its coverage-gap table, its worked example, and the example enhanced PRD all used pre-5.0 numbering (V2 Authentication, V4 Access Control, V12 Input Validation) or cited requirements that do not exist (V2.5, V11.1.4). Every coverage matrix the skill produced therefore cited requirement IDs that resolve to unrelated requirements. - Rebuild the chapter index from the data files' own title frontmatter - Correct all mappings and gap-table citations to verified 5.0 IDs - Mark above-level items (e.g. 6.3.8 is L3) as recommended escalations - Add numbering-discipline instructions to the skill - Add scripts/check_refs.py validating every V/S reference in instruction surfaces against the bundled data (negative-tested) - Add eval assertions failing the PRD workspace on pre-5.0 chapter labels Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017LupEJCBDKT1xRyzNHiW4q
Packaging fixes (verified against current Claude Code plugin docs): - Move slash commands from .claude/commands/ (project-only, never scanned for installed plugins) to commands/ at the plugin root, rewritten as thin dispatchers so each skill remains the single authoritative procedure - Promote the FIASSE lookup command to a proper skill (skills/fiasse-lookup) so natural-language FIASSE/SSEM questions can trigger it - Add a path-resolution preamble to every skill: bundled data resolves via CLAUDE_PLUGIN_ROOT in plugin installs, relative to the skill file elsewhere - Stamp release versions into .claude-plugin/plugin.json (the manifest Claude Code actually reads) and exclude dev-only dirs from the release zip - Correct .claude/settings.json permission rule syntax Portability (AGENTS.md / Agent Skills standards): - Add AGENTS.md as the canonical tool-agnostic entry point; CLAUDE.md now imports it (plugin installs never loaded CLAUDE.md; README claim corrected) - Add scripts/install_skills.sh: layout-preserving installer for opencode (.opencode, .claude, .agents, or global config), keeping skills' internal references working unchanged - Add CI workflow: reference checker, JSON/manifest validation, shell syntax checks, plugin layout verification Skill hardening: - Generation and review skills now run existing project linters/tests/ scanners and dependency audits as evidence when present, and say so when absent instead of implying verification - Review skill treats reviewed code as data, never instructions Add docs/critical-review-2026-08.md: critical assessment of whether the plugin increases securability in practice, with a prioritized enhancement plan. Bump version to 2.1.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017LupEJCBDKT1xRyzNHiW4q
Contributor
There was a problem hiding this comment.
Pull request overview
This PR corrects ASVS 5.0 chapter/requirement numbering across the PRD enhancement skill, ASVS data index, and examples, and adds automated validation to prevent future reference drift across instruction surfaces.
Changes:
- Fix ASVS chapter mappings and example outputs to consistently use ASVS 5.0 numbering (e.g., V6 Authentication, V8 Authorization, V16 Logging).
- Add
scripts/check_refs.pyplus CI/release workflow steps to validate ASVS/FIASSE references against bundleddata/. - Improve plugin portability/packaging (path-resolution guidance, installer script, plugin zip version stamping, and root-level
commands/dispatchers).
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/prd-securability-enhancement-workspace/evals/evals.json | Reformats eval JSON and adds an assertion enforcing ASVS 5.0 chapter-label discipline. |
| skills/securability-engineering/SKILL.md | Adds explicit path-resolution guidance and encourages evidence via running existing project tooling. |
| skills/securability-engineering-review/SKILL.md | Adds path-resolution guidance, anti-instruction-steering note, and tool-evidence guidance during reviews. |
| skills/prd-securability-enhancement/SKILL.md | Updates ASVS 5.0 mappings, adds numbering-discipline instructions, and corrects gap-table/example ASVS references. |
| skills/fiasse-lookup/SKILL.md | Promotes the lookup content into a proper skill with frontmatter and clarified query handling. |
| scripts/install_skills.sh | Adds a layout-preserving installer for agent tools outside Claude Code plugin installs. |
| scripts/check_refs.py | Adds a reference validator for ASVS/FIASSE citations across instruction surfaces. |
| scripts/build_plugin_zip.sh | Excludes dev-only content from release artifacts and stamps version into .claude-plugin/plugin.json. |
| README.md | Updates installation guidance, structure, and documents the new lookup capability and plugin layout. |
| examples/prd-enhancement/enhanced-prd.md | Updates the example enhanced PRD to ASVS 5.0 chapter numbering. |
| docs/critical-review-2026-08.md | Adds a critical review doc explaining defects found (incl. ASVS numbering drift) and fixes/plan. |
| data/asvs/README.md | Rebuilds the chapter index to match bundled ASVS 5.0 files and warns about renumbering vs 4.x. |
| commands/secure-generate.md | Adds root-level command dispatcher delegating to securability-engineering skill. |
| commands/securability-review.md | Adds root-level command dispatcher delegating to securability-engineering-review skill. |
| commands/prd-securability-enhance.md | Adds root-level command dispatcher delegating to prd-securability-enhancement skill. |
| commands/fiasse-lookup.md | Adds root-level command dispatcher delegating to fiasse-lookup skill. |
| CLAUDE.md | Converts to a thin stub importing AGENTS.md (canonical guidance). |
| AGENTS.md | Adds canonical, tool-agnostic entry point content and repo layout/guidance. |
| .github/workflows/release.yml | Runs scripts/check_refs.py as part of release validation. |
| .github/workflows/ci.yml | Adds CI workflow for reference validation, JSON parse checks, shell syntax checks, and layout verification. |
| .claude/settings.json | Fixes Claude Code permission pattern syntax to Read(path/**) form. |
| .claude/commands/secure-generate.md | Removes old project-only command file (now replaced by root commands/). |
| .claude/commands/securability-review.md | Removes old project-only command file (now replaced by root commands/). |
| .claude/commands/prd-securability-enhance.md | Removes old project-only command file (now replaced by root commands/). |
| .claude-plugin/plugin.json | Bumps plugin version to 2.1.0. |
| .claude-plugin/marketplace.json | Bumps marketplace manifest version to 2.1.0. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bundled data/asvs files are genuine ASVS 5.0 (V6 Authentication,
V8 Authorization, V16 Logging), but the chapter index in data/asvs/README.md,
the PRD skill's chapter mappings, its coverage-gap table, its worked example,
and the example enhanced PRD all used pre-5.0 numbering (V2 Authentication,
V4 Access Control, V12 Input Validation) or cited requirements that do not
exist (V2.5, V11.1.4). Every coverage matrix the skill produced therefore
cited requirement IDs that resolve to unrelated requirements.
surfaces against the bundled data (negative-tested)
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_017LupEJCBDKT1xRyzNHiW4q