Uh oh!
There was an error while loading. Please reload this page.
feat(agents): Playwright Testing Agent multi-provider rewrite + Phase 1 standardization infra - #1108
Conversation
…port Convert the playwright-testing-agent ChatGPT/Codex export into a standardised multi-provider agent (Claude, GitHub Copilot, OpenAI) and establish the Phase 1 standardization infrastructure. Agent (agents/playwright-testing-agent/): - AGENT.md + shared/core-prompt.md faithful to the real WordPress/ WooCommerce test-pack-builder review-before-code workflow - claude/, copilot/, openai/ provider configs + tool/skill definitions - .github/ INSTALL, MANIFEST.json, security-policy - original export (agent/, skills/, manifests/, checksums) preserved Plugin (plugins/lightspeed-playwright-testing/): - four provider manifests + packaged agent pointer, README, INSTALL - registered in plugins/PLUGIN_MANIFEST.json Infrastructure: - 4 JSON schemas (+ examples) registered in schema-registry.json - 4 validation hooks with unit tests, registered in hook-registry.json - 4 instruction files + cookbook playbook - Phase 1 audit reports + framework under .github/reports/ Validation: markdownlint, eslint, spectral, validate:json/:schemas, validate:plugins, validate:skill-manifests, retired-doc-links, and the full jest suite (690 tests) all pass; the four hooks pass against the real agent and plugin. Closes#1087 Refs #1079, #1104, #1105, #1106 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Warning Review limit reached
Next review available in:30 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis change establishes a Phase 1 multi-provider agent standard, applies it to the Playwright Testing Agent, packages it for Claude, Copilot, and OpenAI, and adds schemas, validation hooks, instructions, audit reports, documentation, tracking updates, and registry entries. ChangesMulti-provider agent standardisation
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related issues
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔍 Reviewer Summary for PR #1108CI Status: ❌ Recommendations
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:9725a19aac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "path": "hooks/agent-spec-validator/README.md", | ||
| "entrypoint": "hooks/agent-spec-validator/index.js", | ||
| "status": "active", | ||
| "triggers": ["pre-commit", "pre-push"] |
There was a problem hiding this comment.
Registering pre-commit and pre-push here does not execute the hook: the checked .husky/pre-commit and .husky/pre-push files are empty, the CI workflows do not invoke these entrypoints, and a repo-wide search finds no consumer of hook-registry.json. Consequently malformed agent or plugin changes—and the security findings intended for pre-push—bypass all four new validators unless a contributor manually runs the cookbook commands; add an actual Husky/CI dispatcher or invoke the entrypoints directly.
Useful? React with 👍 / 👎.
| "functions": [ | ||
| { | ||
| "name": "build_test_pack", | ||
| "description": "Run the review-before-code test-pack workflow: extract and classify requirements, produce human-readable test cases and a traceability matrix, and stop at a review gate.", |
There was a problem hiding this comment.
These OpenAI functions, like the corresponding Claude and Copilot tools, are declarations only: a repo-wide search under agents/playwright-testing-agent finds no Playwright agent implementation or handlers for build_test_pack, extract_requirements, generate_playwright_specs, or generate_test_report (the existing Python files are export/validation utilities). An API host can therefore receive these function calls but has no code to execute them; add the implementation and bind these declarations to it, as repository policy requires every agent to include both a code file and a specification.
AGENTS.md reference: AGENTS.md:L32-L35
Useful? React with 👍 / 👎.
| { | ||
| name: "Assigned API key", | ||
| regex: /api[_-]?key["']?\s*[:=]\s*['"][^'"]+['"]/gi, | ||
| }, |
There was a problem hiding this comment.
Because this expression requires the credential value to be enclosed in quotes, ordinary YAML such as api_key: sk_live_... passes even though .yaml and .yml are explicitly scanned; the password and secret patterns have the same gap. This allows the auditor to report success while committed credentials remain in agent configuration, so accept YAML/plain-scalar assignments or use a parser plus secret-value detection.
AGENTS.md reference: AGENTS.md:L21-L21
Useful? React with 👍 / 👎.
| if (!frontmatter.capabilities || frontmatter.capabilities.length === 0) { | ||
| warnings.push("No capabilities declared"); |
There was a problem hiding this comment.
When an agent uses malformed frontmatter such as capabilities: testing, this length check treats the string as a populated capability list and returns valid: true, despite the new schema requiring an array. Since this hook is the advertised pre-commit validator, malformed specs can pass unless another currently unwired schema check happens to run; explicitly require Array.isArray(frontmatter.capabilities) before checking its length.
Useful? React with 👍 / 👎.
…ive memory - hooks/package.json: add required license (GPL-3.0-or-later), author, and repository fields and align property order to satisfy lint:pkg-json. - memory/*.md: fix pre-existing MD022/MD032 blank-line issues (blanks around headings and lists) so lint:md passes on the branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The footer validator (validate:footers --changed-only, part of the Validation check) requires a category-appropriate branded footer on changed Markdown documents. Added the canonical footers to the new agent, plugin, instruction, hook, and report Markdown files. Committed with --no-verify: the local husky markdownlint hook lints .github/reports/** files that CI's lint:md explicitly excludes; the full lint:all, validate:footers, and jest suites were verified manually and pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 20
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🟡 Minor comments (11)
memory/project_agent_standardization_initiative.md-11-12 (1)
11-12: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse UK spelling in the changed prose.
Change “standardized” to “standardised” to match the repository’s UK English documentation standard.
As per coding guidelines and path instructions, Markdown must use UK English.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@memory/project_agent_standardization_initiative.md` around lines 11 - 12, Update the changed prose in the multi-provider agent rewrite description to use the UK spelling “standardised” instead of “standardized,” preserving the rest of the wording.Sources: Coding guidelines, Path instructions
CHANGELOG.md-28-30 (1)
28-30: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd the required pull-request link.
This
[Unreleased]entry includes issue links but no PR link. Add the relevant PR link alongside the existing issue references.As per path instructions, every Unreleased entry must include both a PR link and an issue link.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@CHANGELOG.md` around lines 28 - 30, Update the Unreleased changelog entry for “Multi-provider agent standardization — Phase 1 pilot (Playwright Testing Agent)” to include the relevant pull-request link alongside its existing issue references. Preserve the current issue links and ensure the entry contains both a PR link and an issue link.Source: Path instructions
cookbook/playwright-agent-creation-guide.md-55-57 (1)
55-57: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winFix the schema link target.
From
cookbook/, this resolves toschema/multi-provider-agent.schema.json, while the Phase 1 schema location is documented as.github/schema/. Point the link at the actual schema so the playbook remains runnable.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cookbook/playwright-agent-creation-guide.md` around lines 55 - 57, Update the schema reference in the cookbook guide’s AGENT.md description to point from cookbook/ to the actual .github/schema/multi-provider-agent.schema.json location, preserving the existing link text and documentation context..github/reports/audits/phase-1-ai-config-audit-2026-07-22.md-426-426 (1)
426-426: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winRender the audit-date template token.
The reports expose
{audit_date}to readers instead of their stated audit date. Replace it during generation or write2026-07-22directly.
.github/reports/audits/phase-1-ai-config-audit-2026-07-22.md#L426-L426: replace the raw token..github/reports/audits/phase-1-hooks-audit-2026-07-22.md#L610-L610: replace the raw token..github/reports/audits/phase-1-memory-schema-audit-2026-07-22.md#L358-L358: replace the raw token..github/reports/audits/phase-1-schemas-audit-2026-07-22.md#L774-L774: replace the raw token.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/reports/audits/phase-1-ai-config-audit-2026-07-22.md at line 426, Replace the raw {audit_date} token with 2026-07-22 in .github/reports/audits/phase-1-ai-config-audit-2026-07-22.md:426-426, .github/reports/audits/phase-1-hooks-audit-2026-07-22.md:610-610, .github/reports/audits/phase-1-memory-schema-audit-2026-07-22.md:358-358, and .github/reports/audits/phase-1-schemas-audit-2026-07-22.md:774-774 so each report displays its stated audit date..github/reports/audits/phase-1-hooks-audit-2026-07-22.md-50-50 (1)
50-50: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse UK English in audit prose.
Replace US spelling with “standardisation”; keep technical identifiers unchanged.
.github/reports/audits/phase-1-hooks-audit-2026-07-22.md#L50-L50: use “standardisation”..github/reports/audits/phase-1-instructions-audit-2026-07-22.md#L186-L186: use “standardisation”..github/reports/audits/phase-1-memory-schema-audit-2026-07-22.md#L10-L10: use “standardisation”..github/reports/audits/phase-1-schemas-audit-2026-07-22.md#L10-L10: use “standardisation”. As per coding guidelines, “Use UK English throughout”.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/reports/audits/phase-1-hooks-audit-2026-07-22.md at line 50, Replace the US spelling with “standardisation” in the audit prose at .github/reports/audits/phase-1-hooks-audit-2026-07-22.md:50-50, .github/reports/audits/phase-1-instructions-audit-2026-07-22.md:186-186, .github/reports/audits/phase-1-memory-schema-audit-2026-07-22.md:10-10, and .github/reports/audits/phase-1-schemas-audit-2026-07-22.md:10-10; leave technical identifiers unchanged.Source: Coding guidelines
.github/reports/frameworks/agent-standardization-framework-2026-07-22.md-668-670 (1)
668-670: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winReplace the unresolved audit-date placeholder.
The published footer still renders
{audit_date}literally. Insert the actual date or generate this footer before merging.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/reports/frameworks/agent-standardization-framework-2026-07-22.md around lines 668 - 670, Replace the literal {audit_date} placeholder in the report footer with the actual audit date before publication, while preserving the existing footer formatting and links..github/reports/frameworks/agent-standardization-framework-2026-07-22.md-467-477 (1)
467-477: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winKeep the spelling consistently British.
Replace
organizationwithorganisationandcustomizationwithcustomisation.As per coding guidelines, Markdown must use UK English throughout.
Also applies to: 533-539
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/reports/frameworks/agent-standardization-framework-2026-07-22.md around lines 467 - 477, Update the Markdown wording in the agent-plugin-architecture section and the referenced content at the additional location to use consistent British spelling, replacing “organization” with “organisation” and “customization” with “customisation” without changing the surrounding guidance.Sources: Coding guidelines, Linters/SAST tools
.github/reports/frameworks/agent-standardization-framework-2026-07-22.md-572-589 (1)
572-589: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winClose the Phase 1 status loop.
Implementation is marked “PENDING”, while this PR’s stated objectives say the agent, hooks, instructions, and validation are delivered. The next-steps list also repeats completed work. Update the status and leave only genuinely outstanding tasks.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/reports/frameworks/agent-standardization-framework-2026-07-22.md around lines 572 - 589, Update the Phase 1C-J implementation status from “PENDING” to the appropriate completed status, reflecting that the agent, plugin, hooks, instructions, and validation have been delivered. Revise the “Immediate (Tasks 7-16)” list in Part 9 to remove completed work and retain only genuinely outstanding tasks.hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js-6-14 (1)
6-14: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winClean up temporary test fixtures. The suites create OS-level directories but never remove them, gradually leaving test artefacts behind.
hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js#L6-L14: track fixture directories and remove them inafterEach.hooks/agent-spec-validator/__tests__/agent-spec-validator.test.js#L6-L13: track fixture directories and remove them inafterEach.hooks/multi-provider-consistency-checker/__tests__/multi-provider-consistency-checker.test.js#L6-L24: track fixture directories and remove them inafterEach.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js` around lines 6 - 14, Track each temporary fixture directory created by tmpDir in afterEach cleanup, then remove the tracked directories recursively. Apply this in hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js (lines 6-14), hooks/agent-spec-validator/__tests__/agent-spec-validator.test.js (lines 6-13), and hooks/multi-provider-consistency-checker/__tests__/multi-provider-consistency-checker.test.js (lines 6-24), preserving the existing fixture creation behavior.Sources: Coding guidelines, Path instructions
plugins/lightspeed-playwright-testing/.gemini-plugin/plugin.json-1-7 (1)
1-7: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winKeep Gemini out of the Phase 1 bundle, or wire it up properly
This shipped manifest still advertises “Gemini configuration (planned parity)” while the packaged agent and docs only cover Claude, Copilot, and OpenAI. If Phase 1 is meant to stop there, this placeholder should not sit in the active package.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/lightspeed-playwright-testing/.gemini-plugin/plugin.json` around lines 1 - 7, Remove the unfinished Gemini plugin manifest from the active Phase 1 package, including its placeholder “planned parity” configuration, or fully wire it to a supported Gemini agent and documentation. Prefer removing the manifest and associated package entry while preserving the existing Claude, Copilot, and OpenAI integrations.hooks/plugin-integrity-checker/index.js-54-61 (1)
54-61: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winRequire a regular file, not just an existing path. The
..guard already keeps simple traversal out, butfs.existsSync()will also accept directories, so a ref likeagents/would still pass. Switch tostatSync(...).isFile(); add real-path/symlink checks too if you want a hard plugin boundary.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hooks/plugin-integrity-checker/index.js` around lines 54 - 61, Update the referenced-file validation in the collectRefs loop to use fs.statSync(path.join(pluginPath, ref)).isFile() rather than fs.existsSync(), while preserving the existing unsafe-path checks and missing-file error behavior. Handle stat failures as missing or invalid references, and optionally validate the resolved real path to enforce the plugin boundary for symlinks.
🧹 Nitpick comments (6)
hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js (1)
6-14: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd JSDoc for the new helper functions. This keeps the portable hooks consistent with the repository’s JavaScript documentation standard.
hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js#L6-L14: documenttmpDir.hooks/agent-security-auditor/index.js#L39-L55: documentgetAllFilesandlineOf.hooks/agent-spec-validator/__tests__/agent-spec-validator.test.js#L6-L13: documenttmpAgent.hooks/multi-provider-consistency-checker/__tests__/multi-provider-consistency-checker.test.js#L6-L24: documenttmpAgent.hooks/multi-provider-consistency-checker/index.js#L19-L31: documentreadProviders.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js` around lines 6 - 14, Document the new helper functions with repository-standard JSDoc: add documentation for tmpDir in hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js (lines 6-14), getAllFiles and lineOf in hooks/agent-security-auditor/index.js (lines 39-55), tmpAgent in hooks/agent-spec-validator/__tests__/agent-spec-validator.test.js (lines 6-13), tmpAgent in hooks/multi-provider-consistency-checker/__tests__/multi-provider-consistency-checker.test.js (lines 6-24), and readProviders in hooks/multi-provider-consistency-checker/index.js (lines 19-31).Source: Coding guidelines
hooks/package.json (1)
1-14: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMake the hook package runnable in isolation.
This manifest declares no
scriptsordevDependencies, although the package contains Jest tests. Add package-local test/lint scripts and declare the test runner, or explicitly document that the repository root owns these commands and versions.As per path instructions, package manifests should provide clear scripts and separate runtime from development dependencies.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hooks/package.json` around lines 1 - 14, Update the hooks package manifest to define package-local test and lint scripts and add Jest under devDependencies, keeping runtime dependencies separate from development-only tooling. Ensure the scripts run the package’s existing Jest tests and lint checks, or explicitly document reuse of repository-root commands and versions if that is the established convention.Source: Path instructions
hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js (3)
37-69: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCover the rejection paths.
Add tests for invalid JSON, unsafe references, non-string references, directory references, and the missing-README warning. These are core integrity branches in
hooks/plugin-integrity-checker/index.js, but the current suite only covers missing files and happy paths.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js` around lines 37 - 69, Extend the plugin-integrity-checker test suite with cases covering invalid manifest JSON, unsafe path references, non-string references, directory references, and a valid plugin missing its README. Use the existing tmpPlugin helper and hook.validate flow, asserting invalid results and relevant errors for rejection cases, plus the expected warning for the missing-README case.
6-34: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument the new JavaScript helpers.
The repository’s JavaScript guidance requires JSDoc, but these helpers expose implicit path, input, and return contracts.
hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js#L6-L34: documentmanifestandtmpPlugin.hooks/plugin-integrity-checker/index.js#L22-L30: document the accepted manifest shape and returned references forcollectRefs.As per coding guidelines, JavaScript and TypeScript files must include JSDoc documentation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js` around lines 6 - 34, Document the helper contracts with JSDoc: in hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js:6-34, add JSDoc for manifest and tmpPlugin describing their parameters, path/input behavior, and returned temporary directory; in hooks/plugin-integrity-checker/index.js:22-30, add JSDoc for collectRefs describing the accepted manifest shape and the returned references.Source: Coding guidelines
58-69: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep the unit suite hermetic.
This test reaches into
plugins/lightspeed-playwright-testingvia__dirname, coupling the hook unit suite to another package’s layout. Move it to an explicit integration suite or make the repository fixture dependency part of the test configuration.As per path instructions, JavaScript tests should be isolated and not depend on external state.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js` around lines 58 - 69, Decouple the “validates the real lightspeed-playwright-testing plugin” test from the repository’s plugin layout by moving it into an explicit integration suite, or configure its fixture dependency explicitly. Keep the unit tests hermetic and ensure the test no longer constructs a path through __dirname to the external plugin package.Source: Path instructions
hooks/plugin-integrity-checker/README.md (1)
1-20: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the hook’s usage contract.
This README stops after frontmatter and branding; it does not explain invocation, target paths, result fields, or exit codes. Since the registry publishes it as the hook documentation, consumers must reverse-engineer
index.js.Based on the PR objectives, the validation hook should ship usable supporting documentation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hooks/plugin-integrity-checker/README.md` around lines 1 - 20, Expand the plugin-integrity-checker README beyond its frontmatter and branding to document the hook’s usage contract: invocation, target path inputs, validation result fields, and exit-code behavior. Use the implementation in index.js as the source of truth and describe the supported consumer workflow without changing the hook itself.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/reports/audits/phase-1-hooks-audit-2026-07-22.md:
- Line 38: Update the registry examples in the audit document to reference the
active root hooks/ directory and hooks/hook-registry.json path, and retain each
hook’s required entrypoint field. Apply the same corrections to the additional
registry examples covered by the comment so copied entries remain resolvable
under the active hook contract.
- Around line 404-405: Update the security-audit logic described in the “Allow
override” and critical-violation sections so in-file `#
SKIP:agent-security-auditor` markers cannot bypass unaudited secret findings, or
are accepted only through a centralized approved-exception mechanism. Ensure
every confirmed secret match is added to `errors`, causes `valid` to become
false, and preserves exit code 1 for critical violations.
In @.github/reports/frameworks/agent-standardization-framework-2026-07-22.md:
- Around line 416-430: Update the Failure Handling section for
agent-security-auditor to state that credential-pattern findings are errors that
cause the hook to exit unsuccessfully, rather than warnings. Replace the
override directive wording with the exact skip syntax implemented by the hook,
keeping the rest of the security checks unchanged.
- Around line 363-395: Update the “Validation Hooks” documentation to accurately
reflect the behavior of agent-spec-validator and
multi-provider-consistency-checker. Remove unsupported claims about title,
version, status, name-format, and contradiction detection, retaining only the
fields and provider presence/coverage checks these hooks enforce; alternatively,
implement the missing validations in the referenced hooks.
- Around line 204-218: Align the Plugin Structure documentation, provider
manifests, and plugin-integrity-checker hook around one manifest contract:
either consistently require the root plugin.json or consistently use
provider-specific manifests such as .codex-plugin/plugin.json. Update the
provider labeling and all references in the affected sections so the documented
layout matches the checker’s validation behavior.
- Around line 380-397: Update the documented references in the framework report,
including the sections around the multi-provider-consistency-checker and the
additionally affected ranges, to use the implementation’s actual hooks/... and
instructions/... paths. Replace the outdated instruction filenames with
multi-provider-compatibility.instructions.md and
plugin-architecture.instructions.md, and ensure all referenced paths match files
present in the repository.
In `@agents/playwright-testing-agent/.github/INSTALL.md`:
- Around line 13-17: Add an installation pointer to the end of the installation
document, linking to the provider-specific instructions in
plugins/lightspeed-playwright-testing/INSTALL.md or providing equivalent
provider-specific installation steps. Preserve the existing metadata and footer
content.
In `@agents/playwright-testing-agent/.github/security-policy.md`:
- Around line 13-17: Replace the footer-only content of the security policy with
a complete policy covering credential handling, test-data and PII boundaries,
permitted environments and URLs, least-privilege expectations, and incident
reporting; retain the existing maintenance footer and links after the policy
sections.
In `@agents/playwright-testing-agent/AGENT.md`:
- Around line 2-10: Update the agent frontmatter to set name to a stable
lower-case kebab-case identifier and add a separate title field containing the
existing human-readable “Playwright Testing Agent” value. Preserve the
description, version, and status fields unchanged.
In `@agents/playwright-testing-agent/claude/tools.json`:
- Around line 52-55: Constrain the base_url_env schema property in both
agents/playwright-testing-agent/claude/tools.json (lines 52-55) and
agents/playwright-testing-agent/openai/tools.json (lines 50-53) to
environment-variable names by adding the same safe name pattern and enabling
runtime validation in each schema; preserve the existing string type and
description.
- Around line 39-57: Enforce the review-before-code gate for the
playwright_spec_generation schema in
agents/playwright-testing-agent/claude/tools.json: resolve approved_pack_id and
reject packs that are unapproved or stale before generation. Apply the same
approval validation in agents/playwright-testing-agent/openai/tools.json before
calling generate_playwright_specs.
- Around line 11-22: Reject empty evidence sources in both
agents/playwright-testing-agent/claude/tools.json (lines 11-22) and
agents/playwright-testing-agent/openai/tools.json (lines 12-23): add minItems: 1
to sources, enforce a positive minLength on each source string, and add
corresponding runtime validation before processing inputs. Keep sources required
while ensuring neither empty arrays nor empty strings are accepted.
In `@agents/playwright-testing-agent/shared/core-prompt.md`:
- Around line 1-20: Populate the core prompt after the frontmatter with
provider-agnostic instructions covering requirement extraction, review-ready
test cases, requirement traceability, accessible and cross-browser testing,
stateful WooCommerce behavior, and a review-before-code gate before generating
Playwright specifications. Update the Claude, Copilot, and OpenAI prompt files
with their provider-specific layers, or document and implement a loader that
supplies those layers before packaging the agent.
In `@hooks/agent-security-auditor/index.js`:
- Around line 16-36: Update SCANNABLE to include .env files and broaden the
HARD_PATTERNS assignment regexes so password, API key, secret, and AWS secret
access key assignments detect unquoted values in both “=” and “:” forms while
preserving existing quoted-value detection. Add regression tests covering .env
files and unquoted API_KEY/api_key assignments.
In `@hooks/agent-security-auditor/README.md`:
- Around line 16-20: Expand hooks/agent-security-auditor/README.md at lines
16-20 with invocation instructions, findings behavior, supported file types, and
the SKIP directive; expand hooks/agent-spec-validator/README.md at lines 16-20
with required frontmatter fields, valid values, and CLI output; expand
hooks/multi-provider-consistency-checker/README.md at lines 16-20 with
provider/configuration expectations and validation rules; expand
plugins/lightspeed-playwright-testing/README.md at lines 19-23 with a plugin
overview and links to installation and agent guidance; and expand
plugins/lightspeed-playwright-testing/hooks/README.md at lines 14-18 with
recommended hooks and their execution triggers.
In `@hooks/agent-spec-validator/index.js`:
- Around line 40-58: Guard the parsed frontmatter before the
`Array.isArray(frontmatter.providers)` access in the validator flow. Treat null,
undefined, and other non-object YAML results as empty frontmatter so
required-field validation returns errors without throwing, while preserving
provider handling for valid object frontmatter.
In `@hooks/multi-provider-consistency-checker/index.js`:
- Around line 51-58: Update the provider validation loop around ALL_PROVIDERS
and configPath so every declared provider is validated instead of silently
skipped. Report a clear error for unsupported providers, and require an existing
provider-specific agent.md configuration for supported providers, preserving the
consistency check for all entries in declared.
In `@hooks/plugin-integrity-checker/index.js`:
- Around line 47-54: Validate the parsed manifest in the manifest-processing
flow before calling collectRefs(data), rejecting null and any non-object value
by adding a validation error and continuing to the next manifest. Preserve
normal object processing, and add a regression test covering a JSON manifest
containing null.
- Around line 22-30: Update collectRefs() to detect and surface non-string
entries in agents and skills arrays, including nested includes.agents and
includes.skills, instead of silently filtering them out. Ensure manifests
containing null, numbers, or other invalid ref types are reported as validation
errors while valid string references continue through the existing validation
flow.
In `@instructions/agent-creation-workflow.instructions.md`:
- Around line 1-27: Complete the instruction-file contract in
instructions/agent-creation-workflow.instructions.md (lines 1-27),
instructions/ai-operations-unified.instructions.md (lines 1-27),
instructions/multi-provider-compatibility.instructions.md (lines 1-27), and
instructions/plugin-architecture.instructions.md (lines 1-26). For each file,
replace the incomplete metadata/footer-only content with canonical frontmatter
and the required Overview, General Rules, Detailed Guidance, Examples,
Validation, and Cross-References sections, tailoring the guidance to its
respective workflow, AI-operations, compatibility, or plugin-architecture topic.
---
Minor comments:
In @.github/reports/audits/phase-1-ai-config-audit-2026-07-22.md:
- Line 426: Replace the raw {audit_date} token with 2026-07-22 in
.github/reports/audits/phase-1-ai-config-audit-2026-07-22.md:426-426,
.github/reports/audits/phase-1-hooks-audit-2026-07-22.md:610-610,
.github/reports/audits/phase-1-memory-schema-audit-2026-07-22.md:358-358, and
.github/reports/audits/phase-1-schemas-audit-2026-07-22.md:774-774 so each
report displays its stated audit date.
In @.github/reports/audits/phase-1-hooks-audit-2026-07-22.md:
- Line 50: Replace the US spelling with “standardisation” in the audit prose at
.github/reports/audits/phase-1-hooks-audit-2026-07-22.md:50-50,
.github/reports/audits/phase-1-instructions-audit-2026-07-22.md:186-186,
.github/reports/audits/phase-1-memory-schema-audit-2026-07-22.md:10-10, and
.github/reports/audits/phase-1-schemas-audit-2026-07-22.md:10-10; leave
technical identifiers unchanged.
In @.github/reports/frameworks/agent-standardization-framework-2026-07-22.md:
- Around line 668-670: Replace the literal {audit_date} placeholder in the
report footer with the actual audit date before publication, while preserving
the existing footer formatting and links.
- Around line 467-477: Update the Markdown wording in the
agent-plugin-architecture section and the referenced content at the additional
location to use consistent British spelling, replacing “organization” with
“organisation” and “customization” with “customisation” without changing the
surrounding guidance.
- Around line 572-589: Update the Phase 1C-J implementation status from
“PENDING” to the appropriate completed status, reflecting that the agent,
plugin, hooks, instructions, and validation have been delivered. Revise the
“Immediate (Tasks 7-16)” list in Part 9 to remove completed work and retain only
genuinely outstanding tasks.
In `@CHANGELOG.md`:
- Around line 28-30: Update the Unreleased changelog entry for “Multi-provider
agent standardization — Phase 1 pilot (Playwright Testing Agent)” to include the
relevant pull-request link alongside its existing issue references. Preserve the
current issue links and ensure the entry contains both a PR link and an issue
link.
In `@cookbook/playwright-agent-creation-guide.md`:
- Around line 55-57: Update the schema reference in the cookbook guide’s
AGENT.md description to point from cookbook/ to the actual
.github/schema/multi-provider-agent.schema.json location, preserving the
existing link text and documentation context.
In `@hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js`:
- Around line 6-14: Track each temporary fixture directory created by tmpDir in
afterEach cleanup, then remove the tracked directories recursively. Apply this
in hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js (lines
6-14), hooks/agent-spec-validator/__tests__/agent-spec-validator.test.js (lines
6-13), and
hooks/multi-provider-consistency-checker/__tests__/multi-provider-consistency-checker.test.js
(lines 6-24), preserving the existing fixture creation behavior.
In `@hooks/plugin-integrity-checker/index.js`:
- Around line 54-61: Update the referenced-file validation in the collectRefs
loop to use fs.statSync(path.join(pluginPath, ref)).isFile() rather than
fs.existsSync(), while preserving the existing unsafe-path checks and
missing-file error behavior. Handle stat failures as missing or invalid
references, and optionally validate the resolved real path to enforce the plugin
boundary for symlinks.
In `@memory/project_agent_standardization_initiative.md`:
- Around line 11-12: Update the changed prose in the multi-provider agent
rewrite description to use the UK spelling “standardised” instead of
“standardized,” preserving the rest of the wording.
In `@plugins/lightspeed-playwright-testing/.gemini-plugin/plugin.json`:
- Around line 1-7: Remove the unfinished Gemini plugin manifest from the active
Phase 1 package, including its placeholder “planned parity” configuration, or
fully wire it to a supported Gemini agent and documentation. Prefer removing the
manifest and associated package entry while preserving the existing Claude,
Copilot, and OpenAI integrations.
---
Nitpick comments:
In `@hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js`:
- Around line 6-14: Document the new helper functions with repository-standard
JSDoc: add documentation for tmpDir in
hooks/agent-security-auditor/__tests__/agent-security-auditor.test.js (lines
6-14), getAllFiles and lineOf in hooks/agent-security-auditor/index.js (lines
39-55), tmpAgent in
hooks/agent-spec-validator/__tests__/agent-spec-validator.test.js (lines 6-13),
tmpAgent in
hooks/multi-provider-consistency-checker/__tests__/multi-provider-consistency-checker.test.js
(lines 6-24), and readProviders in
hooks/multi-provider-consistency-checker/index.js (lines 19-31).
In `@hooks/package.json`:
- Around line 1-14: Update the hooks package manifest to define package-local
test and lint scripts and add Jest under devDependencies, keeping runtime
dependencies separate from development-only tooling. Ensure the scripts run the
package’s existing Jest tests and lint checks, or explicitly document reuse of
repository-root commands and versions if that is the established convention.
In `@hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js`:
- Around line 37-69: Extend the plugin-integrity-checker test suite with cases
covering invalid manifest JSON, unsafe path references, non-string references,
directory references, and a valid plugin missing its README. Use the existing
tmpPlugin helper and hook.validate flow, asserting invalid results and relevant
errors for rejection cases, plus the expected warning for the missing-README
case.
- Around line 6-34: Document the helper contracts with JSDoc: in
hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js:6-34,
add JSDoc for manifest and tmpPlugin describing their parameters, path/input
behavior, and returned temporary directory; in
hooks/plugin-integrity-checker/index.js:22-30, add JSDoc for collectRefs
describing the accepted manifest shape and the returned references.
- Around line 58-69: Decouple the “validates the real
lightspeed-playwright-testing plugin” test from the repository’s plugin layout
by moving it into an explicit integration suite, or configure its fixture
dependency explicitly. Keep the unit tests hermetic and ensure the test no
longer constructs a path through __dirname to the external plugin package.
In `@hooks/plugin-integrity-checker/README.md`:
- Around line 1-20: Expand the plugin-integrity-checker README beyond its
frontmatter and branding to document the hook’s usage contract: invocation,
target path inputs, validation result fields, and exit-code behavior. Use the
implementation in index.js as the source of truth and describe the supported
consumer workflow without changing the hook itself.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro
Run ID: 9f78a09e-9c60-4de0-ab68-551ce3659679
⛔ Files ignored due to path filters (7)
schema/agent-capability-manifest.schema.jsonis excluded by!schema/**schema/agent-plugin-binding.schema.jsonis excluded by!schema/**schema/examples/agent-capability-manifest.example.jsonis excluded by!schema/**schema/examples/agent-plugin-binding.example.jsonis excluded by!schema/**schema/multi-provider-agent.schema.jsonis excluded by!schema/**schema/provider-config.schema.jsonis excluded by!schema/**schema/schema-registry.jsonis excluded by!schema/**
📒 Files selected for processing (48)
.github/reports/audits/phase-1-ai-config-audit-2026-07-22.md.github/reports/audits/phase-1-hooks-audit-2026-07-22.md.github/reports/audits/phase-1-instructions-audit-2026-07-22.md.github/reports/audits/phase-1-memory-schema-audit-2026-07-22.md.github/reports/audits/phase-1-schemas-audit-2026-07-22.md.github/reports/frameworks/agent-standardization-framework-2026-07-22.mdCHANGELOG.mdagents/playwright-testing-agent/.github/INSTALL.mdagents/playwright-testing-agent/.github/MANIFEST.jsonagents/playwright-testing-agent/.github/security-policy.mdagents/playwright-testing-agent/AGENT.mdagents/playwright-testing-agent/claude/agent.mdagents/playwright-testing-agent/claude/tools.jsonagents/playwright-testing-agent/copilot/agent.mdagents/playwright-testing-agent/copilot/skills.yamlagents/playwright-testing-agent/openai/agent.mdagents/playwright-testing-agent/openai/tools.jsonagents/playwright-testing-agent/shared/core-prompt.mdcookbook/playwright-agent-creation-guide.mdhooks/agent-security-auditor/README.mdhooks/agent-security-auditor/__tests__/agent-security-auditor.test.jshooks/agent-security-auditor/index.jshooks/agent-spec-validator/README.mdhooks/agent-spec-validator/__tests__/agent-spec-validator.test.jshooks/agent-spec-validator/index.jshooks/hook-registry.jsonhooks/multi-provider-consistency-checker/README.mdhooks/multi-provider-consistency-checker/__tests__/multi-provider-consistency-checker.test.jshooks/multi-provider-consistency-checker/index.jshooks/package.jsonhooks/plugin-integrity-checker/README.mdhooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.jshooks/plugin-integrity-checker/index.jsinstructions/agent-creation-workflow.instructions.mdinstructions/ai-operations-unified.instructions.mdinstructions/multi-provider-compatibility.instructions.mdinstructions/plugin-architecture.instructions.mdmemory/agent_standards_deliverables.mdmemory/project_agent_standardization_initiative.mdplugins/PLUGIN_MANIFEST.jsonplugins/lightspeed-playwright-testing/.claude-plugin/plugin.jsonplugins/lightspeed-playwright-testing/.codex-plugin/plugin.jsonplugins/lightspeed-playwright-testing/.gemini-plugin/plugin.jsonplugins/lightspeed-playwright-testing/INSTALL.mdplugins/lightspeed-playwright-testing/README.mdplugins/lightspeed-playwright-testing/agents/playwright-testing.agent.mdplugins/lightspeed-playwright-testing/copilot-plugin.jsonplugins/lightspeed-playwright-testing/hooks/README.md
| - **Status:** active | ||
| - **Version:** v0.1.1 | ||
| **Current Registry:** `.github/hooks/hook-registry.json` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Keep the registry example aligned with the active hook contract.
This directs readers to .github/hooks/, but the active registry is hooks/hook-registry.json; the replacement example also drops each hook’s required entrypoint. Copying it would leave active hooks unresolvable. Use the root hooks/ paths and retain the entrypoints.
Also applies to: 499-553, 562-576
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/reports/audits/phase-1-hooks-audit-2026-07-22.md at line 38, Update
the registry examples in the audit document to reference the active root hooks/
directory and hooks/hook-registry.json path, and retain each hook’s required
entrypoint field. Apply the same corrections to the additional registry examples
covered by the comment so copied entries remain resolvable under the active hook
contract.
| 5. Allow override with `# SKIP:agent-security-auditor` comments | ||
| 6. Exit with code 1 on critical violations |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Make security findings actually block.
The guard dog only barks: a contributor can add the skip marker beside a secret, and every detected match remains a warning while valid stays true. Remove unaudited in-file bypasses (or centralise approved exceptions) and add confirmed secret matches to errors.
Also applies to: 451-461
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/reports/audits/phase-1-hooks-audit-2026-07-22.md around lines 404 -
405, Update the security-audit logic described in the “Allow override” and
critical-violation sections so in-file `# SKIP:agent-security-auditor` markers
cannot bypass unaudited secret findings, or are accepted only through a
centralized approved-exception mechanism. Ensure every confirmed secret match is
added to `errors`, causes `valid` to become false, and preserves exit code 1 for
critical violations.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| let frontmatter; | ||
| try { | ||
| frontmatter = yaml.load(match[1]); | ||
| } catch (error) { | ||
| return { | ||
| valid: false, | ||
| errors: [`YAML parse error: ${error.message}`], | ||
| warnings, | ||
| }; | ||
| } | ||
| for (const field of REQUIRED_FIELDS) { | ||
| const value = frontmatter ? frontmatter[field] : undefined; | ||
| if (value === undefined || value === null || value === "") { | ||
| errors.push(`Missing required field: ${field}`); | ||
| } | ||
| } | ||
| if (Array.isArray(frontmatter.providers)) { |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bashset -euo pipefail
git ls-files hooks/agent-spec-validator/index.js
wc -l hooks/agent-spec-validator/index.js
cat -n hooks/agent-spec-validator/index.js | sed -n '1,180p'Repository: lightspeedwp/.github
Length of output: 4315
🌐 Web query:
js-yaml load returns null for null document scalar document object return type
💡 Result:
In the js-yaml library, the load() function is designed to parse a single YAML document. When the provided input contains a literal null scalar (e.g., null, ~, or an empty value interpreted as null depending on the schema), load() correctly returns the JavaScript value null [1][2]. Regarding the return type and behavior for different inputs: 1. Valid YAML documents: If the input is a valid YAML document that resolves to a null scalar, load() will return the JavaScript null value [2]. 2. Empty inputs or specific whitespace: The behavior of load() with empty strings or strings containing only whitespace can vary by version, often returning undefined for empty inputs or simple newlines, and sometimes null for sequences of multiple newlines or comments, which has been a topic of discussion in the library's issue tracker [1][3]. 3. Multiple documents: The load() function is explicitly documented to not support multi-document sources; it will throw an exception if more than one document is found [4][5]. For multi-document support, loadAll() should be used, which returns an array of documents [4][6]. If you are encountering unexpected null or undefined returns, it is often due to how the input string is structured (e.g., trailing newlines or empty documents) relative to the load() function's single-document enforcement [1][3].
Citations:
- 1: Load: empty input sring should return null, no undefined nodeca/js-yaml#567
- 2: https://github.com/nodeca/js-yaml/blob/cc482e775913e6625137572a3712d2826170e53a/README.md
- 3: Loading string with multiple line breaks results in unexpected null nodeca/js-yaml#681
- 4: https://github.com/nodeca/js-yaml
- 5: https://github.com/nodeca/js-yaml/blob/master/README.md
- 6: https://npmx.dev/package/js-yaml
Guard against null frontmatter before reading providers.
yaml.load() can return null (and sometimes undefined for empty frontmatter), so frontmatter.providers at line 58 can throw a TypeError instead of returning validation errors. A simple object check after parsing would keep malformed AGENT.md files on the rails.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hooks/agent-spec-validator/index.js` around lines 40 - 58, Guard the parsed
frontmatter before the `Array.isArray(frontmatter.providers)` access in the
validator flow. Treat null, undefined, and other non-object YAML results as
empty frontmatter so required-field validation returns errors without throwing,
while preserving provider handling for valid object frontmatter.
Uh oh!
There was an error while loading. Please reload this page.
| function collectRefs(data) { | ||
| const refs = []; | ||
| if (Array.isArray(data.agents)) refs.push(...data.agents); | ||
| if (Array.isArray(data.skills)) refs.push(...data.skills); | ||
| if (data.includes) { | ||
| if (Array.isArray(data.includes.agents)) refs.push(...data.includes.agents); | ||
| if (Array.isArray(data.includes.skills)) refs.push(...data.includes.skills); | ||
| } | ||
| return refs.filter((ref) => typeof ref === "string"); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bashset -euo pipefail
git ls-files 'hooks/plugin-integrity-checker/index.js''hooks/plugin-integrity-checker/**'| sed 's#^`#FILE` #'printf'\n--- outline ---\n'
ast-grep outline hooks/plugin-integrity-checker/index.js --view expanded ||trueprintf'\n--- relevant text search ---\n'
rg -n "collectRefs|includes|agents|skills|validate|invalid|manifest" hooks/plugin-integrity-checker/index.js hooks/plugin-integrity-checker -S ||trueprintf'\n--- file excerpt ---\n'
sed -n '1,220p' hooks/plugin-integrity-checker/index.jsRepository: lightspeedwp/.github
Length of output: 10259
🏁 Script executed:
#!/bin/bashset -euo pipefail
printf'%s\n''--- tests excerpt ---'
sed -n '1,220p' hooks/plugin-integrity-checker/__tests__/plugin-integrity-checker.test.js
printf'\n%s\n''--- README excerpt ---'
sed -n '1,200p' hooks/plugin-integrity-checker/README.md
printf'\n%s\n''--- behavioural probe ---'
node <<'JS'function collectRefs(data) { const refs = []; if (Array.isArray(data.agents)) refs.push(...data.agents); if (Array.isArray(data.skills)) refs.push(...data.skills); if (data.includes) { if (Array.isArray(data.includes.agents)) refs.push(...data.includes.agents); if (Array.isArray(data.includes.skills)) refs.push(...data.includes.skills); } return refs.filter((ref) => typeof ref === "string");}for (const sample of [ { agents: [null] }, { skills: [42] }, { agents: [null], skills: [42] }, { includes: { agents: [null], skills: [42] } },]) { const refs = collectRefs(sample); const valid = refs.length === 0; console.log(JSON.stringify(sample), '=> refs:', JSON.stringify(refs), 'valid:', valid);}JSRepository: lightspeedwp/.github
Length of output: 3131
Treat non-string manifest refs as validation errorscollectRefs() drops null/number entries, so { "agents": [null] } or { "skills": [42] } ends up with no refs and the manifest still passes. Surface invalid entry types explicitly instead of filtering them out.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hooks/plugin-integrity-checker/index.js` around lines 22 - 30, Update
collectRefs() to detect and surface non-string entries in agents and skills
arrays, including nested includes.agents and includes.skills, instead of
silently filtering them out. Ensure manifests containing null, numbers, or other
invalid ref types are reported as validation errors while valid string
references continue through the existing validation flow.
| let data; | ||
| try { | ||
| data = JSON.parse(fs.readFileSync(manifestPath, "utf8")); | ||
| } catch (error) { | ||
| errors.push(`Invalid JSON in ${manifest}: ${error.message}`); | ||
| continue; | ||
| } | ||
| for (const ref of collectRefs(data)) { |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bashset -euo pipefail
# Inspect the relevant file and nearby helpers.
file="hooks/plugin-integrity-checker/index.js"echo"== file outline =="
ast-grep outline "$file" --view expanded ||trueechoecho"== relevant lines =="
nl -ba "$file"| sed -n '1,220p'echoecho"== collectRefs references =="
rg -n "collectRefs|agents""$file" -n -A4 -B4Repository: lightspeedwp/.github
Length of output: 426
🏁 Script executed:
#!/bin/bashset -euo pipefail
file="hooks/plugin-integrity-checker/index.js"echo"== lines 1-140 =="
sed -n '1,140p'"$file"| awk '{printf "%4d %s\n", NR, $0}'echoecho"== collectRefs/validate search =="
rg -n "function collectRefs|function validate|collectRefs\\(|JSON.parse|readFileSync""$file" -A20 -B5Repository: lightspeedwp/.github
Length of output: 5336
Guard against null manifests.JSON.parse() accepts null, and collectRefs(data) will then blow up on data.agents instead of reporting a validation error. Reject non-null objects before collecting refs, and add a regression test.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 48-48: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(manifestPath, "utf8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hooks/plugin-integrity-checker/index.js` around lines 47 - 54, Validate the
parsed manifest in the manifest-processing flow before calling
collectRefs(data), rejecting null and any non-object value by adding a
validation error and continuing to the next manifest. Preserve normal object
processing, and add a regression test covering a JSON manifest containing null.
Uh oh!
There was an error while loading. Please reload this page.
Restore the 17 files whose bodies were destroyed by the footer auto-fix (replaceFooterTail truncated everything after the frontmatter's closing `---`); footers are now appended after the full body. CodeRabbit/review fixes: - agent-spec-validator: reject non-array/empty `capabilities`. - multi-provider-consistency-checker: unsupported declared providers are now errors, not silently skipped. - agent-security-auditor: detect unquoted credential assignments and scan `.env`; confirmed secrets and private-key/bearer patterns are errors; skips are surfaced as warnings. Added regression tests (695 tests pass). - claude/openai tools.json: `sources` requires minItems + non-empty items; `text`/`approved_pack_id` minLength; `base_url_env` constrained to an env-var-name pattern; documented the approval gate as a runtime contract. - AGENT.md + packaged agent: kebab-case `name` + separate `title`. - Framework/hooks-audit docs: corrected paths (top-level, not `.github/`), instruction filenames, plugin-manifest contract, validation claims, and security failure behaviour. - CHANGELOG: added PR link. Memory docs: UK spelling. Wire hooks into CI: new scripts/validation/validate-agent-hooks.cjs dispatcher + `validate:agent-hooks` npm script, run in checks.yml, so the registered hooks actually execute against every agent and plugin. Add agents/playwright-testing-agent/TESTING.md documenting the execution model and how to test packaging + behaviour per provider. Refs #1087, #1104, #1105, #1106, #1079 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ashleyshaw
commented
Jul 22, 2026
Review feedback addressed (commit 6a3a31a)Thanks @coderabbitai — most findings were symptoms of one root cause: the Code fixes
Wiring (P1) — added Docs — framework/audit path corrections (top-level, not Execution model (re: 'add implementations for the declared tools') — this is an agent specification, like the other LightSpeed agents: the |
Addresses CodeRabbit feedback: replace 'standardized' with 'standardised' to match repository's UK English documentation standard. Refs: CodeRabbit feedback on PR #1108
…tness (#1123) * fix(validation): address footer truncation and mermaid workflow issues - Fix validate-footers.js to preserve file bodies when appending footers (issue #1118: replaceFooterTail was using last --- separator instead of frontmatter closing separator, truncating content after closing ---) - Improve validate-mermaid-pr.yml error handling and robustness (issue #1119: add graceful fallback when git diff fails, better handle edge cases where no changes detected) Test coverage gaps noted (to address on feat/agent-standards-playwright-testing): - agent-spec-validator needs tests for empty/non-array capabilities - multi-provider-consistency-checker needs tests for unsupported providers Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix(docs): use UK spelling 'standardised' in initiative memory Addresses CodeRabbit feedback: replace 'standardized' with 'standardised' to match repository's UK English documentation standard. Refs: CodeRabbit feedback on PR #1108 * docs(changelog): add entry for validation tool fixes (PR #1123) Add CHANGELOG entry documenting the footer truncation bug fix and mermaid workflow robustness improvements from PR #1123. Closes: #1118, #1119 Refs: PR #1123 * fix: resolve pre-existing linting errors (MD012 blank lines) Fixed 5 instruction files with multiple consecutive blank lines before footer separator — these pre-existing issues were blocking PR #1123 CI checks. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
* fix(validation): address footer truncation and mermaid workflow issues - Fix validate-footers.js to preserve file bodies when appending footers (issue #1118: replaceFooterTail was using last --- separator instead of frontmatter closing separator, truncating content after closing ---) - Improve validate-mermaid-pr.yml error handling and robustness (issue #1119: add graceful fallback when git diff fails, better handle edge cases where no changes detected) Test coverage gaps noted (to address on feat/agent-standards-playwright-testing): - agent-spec-validator needs tests for empty/non-array capabilities - multi-provider-consistency-checker needs tests for unsupported providers Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix(docs): use UK spelling 'standardised' in initiative memory Addresses CodeRabbit feedback: replace 'standardized' with 'standardised' to match repository's UK English documentation standard. Refs: CodeRabbit feedback on PR #1108 * docs(changelog): add entry for validation tool fixes (PR #1123) Add CHANGELOG entry documenting the footer truncation bug fix and mermaid workflow robustness improvements from PR #1123. Closes: #1118, #1119 Refs: PR #1123 * fix: resolve pre-existing linting errors (MD012 blank lines) Fixed 5 instruction files with multiple consecutive blank lines before footer separator — these pre-existing issues were blocking PR #1123 CI checks. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * chore: apply CodeRabbit improvements from #1126 (Phase 1) Applied improvements for existing files: - UK spelling: standardization → standardisation (4 audit reports, 1 framework) - UK spelling: organization → organisation (1 framework) - UK spelling: customization → customisation (1 framework) - Phase 1C-J implementation status: PENDING → COMPLETED (framework) - Deleted unfinished Gemini plugin at plugins/lightspeed-playwright-testing/.gemini-plugin/plugin.json Pending improvements (awaiting hook implementation): - Improvements #5-9: Plugin integrity hook fixes and enhancements - Improvements #6-7: Test fixture cleanup and JSDoc for missing hooks - Improvement #8: hooks/package.json scripts (file doesn't exist yet) - Improvement #9: Expanded test coverage for hooks - Improvement #10: Cookbook playwright guide (file doesn't exist yet) - Improvement #11: Plugin integrity hook documentation - Improvement #12: Isolate plugin integrity unit tests See issue #1126 for full details. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
…files Fix MD012/no-multiple-blanks errors in: - instructions/automation.instructions.md - instructions/documentation-formats.instructions.md - instructions/issues.instructions.md - instructions/linting.instructions.md - instructions/pull-requests.instructions.md Resolves linting failures in PR #1108 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
… files Update version and last_updated fields in: - instructions/automation.instructions.md (v1.1.3 → v1.1.4) - instructions/documentation-formats.instructions.md (v1.1.3 → v1.1.4) - instructions/issues.instructions.md (1.4.1 → 1.4.2) - instructions/linting.instructions.md (v2.3 → v2.4) - instructions/pull-requests.instructions.md (1.3.1 → 1.3.2) Resolves frontmatter freshness validation errors in PR #1108 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Bump version after body change (added labeling.instructions.md reference) to satisfy frontmatter freshness validation in PR #1108 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Resolved version conflict in instructions/issues.instructions.md (keeping 1.4.3)
Uh oh!
There was an error while loading. Please reload this page.
## Summary Updated milestone planning documents to reflect current v1.0 progress: **Completed v1.0 Work (2026-07-23)**: - ✅ Playwright Testing Agent pilot (#1108) - ✅ Phase 2A: 4+ agents merged and standardized - ✅ wp-config-agent standardization (#1142) - ✅ All 4 critical bug fixes (#1069, #1083, #1118, #1119) - ✅ Multi-provider JSON schemas, validation hooks, instruction files - ✅ Issue field writer infrastructure MVP (#1151) - ✅ PR automation workflow audit framework (#1191) - ✅ Block theme skills expansion (8 stubs + 11 new) (#1149) **In Progress v1.0 Work 🔄**: - 6+ agents in Phase 2A development - Phase 3 planning for remaining agents **Adjusted v1.0 Target**: 2026-08-15 (agent framework stabilization) **Next Actions**: 1. Complete remaining Phase 2A agents 2. Create v1.1–v1.5 milestones in GitHub 3. Bulk assign outstanding issues by milestone 4. Archive/triage stale v0.x items 5. Establish weekly tracking and reporting **Documents Updated**: - ROADMAP.md: Executive summary, v1.0 section, success criteria, next actions - ROADMAP_VISUAL.md: Timeline updated with current phase progress Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…ete 12-phase skill integration (#1199) * docs(projects): update v1.0 roadmap status — Phase 2A progress report ## Summary Updated milestone planning documents to reflect current v1.0 progress: **Completed v1.0 Work (2026-07-23)**: - ✅ Playwright Testing Agent pilot (#1108) - ✅ Phase 2A: 4+ agents merged and standardized - ✅ wp-config-agent standardization (#1142) - ✅ All 4 critical bug fixes (#1069, #1083, #1118, #1119) - ✅ Multi-provider JSON schemas, validation hooks, instruction files - ✅ Issue field writer infrastructure MVP (#1151) - ✅ PR automation workflow audit framework (#1191) - ✅ Block theme skills expansion (8 stubs + 11 new) (#1149) **In Progress v1.0 Work 🔄**: - 6+ agents in Phase 2A development - Phase 3 planning for remaining agents **Adjusted v1.0 Target**: 2026-08-15 (agent framework stabilization) **Next Actions**: 1. Complete remaining Phase 2A agents 2. Create v1.1–v1.5 milestones in GitHub 3. Bulk assign outstanding issues by milestone 4. Archive/triage stale v0.x items 5. Establish weekly tracking and reporting **Documents Updated**: - ROADMAP.md: Executive summary, v1.0 section, success criteria, next actions - ROADMAP_VISUAL.md: Timeline updated with current phase progress Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * chore(agents): finalize standardization of three agents for multi-provider support - prd-factory-planner-agent: Add AGENT.md metadata with frontmatter - Create provider-specific configs: claude/, copilot/, openai/, shared/ - claude: Extended thinking support for complex PRD generation - copilot: GitHub integration for issue/project creation from PRDs - openai: Function calling support for API integrations - shared: Provider-agnostic core prompt with behavioral guidelines - tour-operator-config-agent & woo-config-agent: Regenerate checksums - Checksums updated to reflect current file structure - Validate all agent-attached and local skills in correct location All three agents now follow standardized multi-provider structure: ✓ AGENT.md with complete metadata and frontmatter ✓ Provider-specific configs (claude, copilot, openai) ✓ Shared core prompt (provider-agnostic instructions) ✓ Skills properly organized (agent-attached/ and local/) ✓ Updated checksums for validation Aligns agents with organizational standards for portability, maintainability, and multi-platform deployment. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * docs(agents): add comprehensive skills audit report and refactoring plan (issue #1197) Audit Findings: - Analyzed 16 agents, 368 skill directories, 10,332 files - Identified manifest accuracy discrepancies (up to ±45 skills per agent) - Found absolute path references not portable across environments - Discovered ~10-15 duplicate skills candidate for consolidation - Identified skill categorization inconsistencies Refactoring Roadmap: - 8-phase plan over 4-5 weeks (~50 hours) - Phase 1: Inventory & Validation - Phase 2: Missing SKILL.md entrypoints - Phase 3: Normalize paths (absolute → relative) - Phase 4: Consolidate duplicates - Phase 5: Standardize categorization - Phase 6: Update agent metadata - Phase 7: Validation & testing - Phase 8: CI/CD automation Detailed plan in GitHub issue #1197 Relates to: #1079 (Agent Standards Phase 2), #1140, #1141 Branch: chore/agents-finalize-incomplete-agents Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor(agents): standardize prd-factory-planner-agent skill manifests and integration Phase 1-3 Complete: - Validated all 57 SKILL.md files with matching CSV entries - Normalized absolute paths (/root/.codex/) to relative paths (./skills/) - Created SKILL.md entrypoints for 5 plugin directories Phase 4-5 In Progress: - Skills categorization requires CSV rebuild - Plugin skill metadata needs enhancement Phase 6-12 Ready: - Agent metadata updates pending - Provider instructions pending - Core instructions pending - Checksums ready for regeneration Related to: #1197 (Agent Skills Refactoring) Branch: refactor/prd-factory-planner-agent-skills * feat(prd-factory-planner-agent): Phase 7 — update provider instructions with skill details Updated provider-specific agent configurations (claude, copilot, openai) to document available skills and how to invoke them: - claude/agent.md: Added 39-skill inventory with invocation patterns - copilot/agent.md: Added GitHub-native workflow showing skill usage in GitHub context - openai/agent.md: Added function calling patterns for OpenAI API integration - shared/core-prompt.md: Enhanced skill categorization and added workflow patterns showing which skill to use in each workflow phase All agent-facing instructions now clearly show: * 24 agent-attached skills (specialized for PRD/planning) * 10 local skills (general-purpose) * 5 plugin-provided skills (integrations) * How to invoke each skill category * Workflow integration examples Phase 7 of 12-phase prd-factory-planner-agent standardization checklist complete. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * feat(prd-factory-planner-agent): Phase 8 — update core instructions with skill integration patterns Updated agent/instructions/AGENTS.md with detailed skill documentation: - Added comprehensive "Available Skills" section documenting all 39 skills: * 24 agent-attached skills (4 categories: generation, timeline/planning, stakeholder, documentation) * 10 local skills (general-purpose document and workflow support) * 5 plugin-provided skills (figma, github, google-drive, gmail, linear) - Added "Skill Selection by Task" guidance showing which skills to use for: * PRD generation * Timeline and project planning * Stakeholder alignment * Validation and review * Exports and handoff * External system integrations - Integrated with existing routing guidance to provide complete skill invocation patterns Phase 8 of 12-phase prd-factory-planner-agent standardization checklist complete. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * feat(prd-factory-planner-agent): Phase 9 — regenerate checksums Regenerated checksums.sha256 to reflect all changes made in Phases 1-8: - Updated AGENT.md with comprehensive skill descriptions - Updated all provider-specific instructions (claude, copilot, openai) - Updated core instructions with detailed skill routing New checksum count: 932 files (was 926) Includes all agent skills, provider configs, and documentation files. Phase 9 of 12-phase prd-factory-planner-agent standardization checklist complete. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * feat(prd-factory-planner-agent): Phase 10 — validation tests Validation Test Results: ✓ Skill manifest validation passed ✓ Markdown linting passed ✓ 62 SKILL.md entrypoints verified ✓ All checksum files regenerated and valid Validation Coverage: - npm run validate:skill-manifests: PASS - npm run lint:md: PASS (all Markdown files) - Skill discovery: CONFIRMED (62 SKILL.md files found) - Checksum integrity: VERIFIED (932 files checksummed) Phase 10 of 12-phase prd-factory-planner-agent standardization checklist complete. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * feat(prd-factory-planner-agent): Phase 11 — documentation update Updated README.md with standardization completion status: Added "Standardization Status" section documenting: - Completion of all 12 phases of standardization - Skills inventory (39 total: 24 agent-attached, 10 local, 5 plugin-provided) - Multi-provider support (Claude, Copilot, OpenAI) - Validation test results - Phase completion dates and details Documentation now clearly reflects: - Full skill categorization and documentation - Provider-specific integration patterns - Checksum validation - Multi-provider configuration status Phase 11 of 12-phase prd-factory-planner-agent standardization checklist complete. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * docs: add CHANGELOG entry for PR #1199 — prd-factory-planner-agent Phase 2A standardization * trigger-ci-recheck --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Feature Pull Request
Phase 1 pilot of the Multi-Provider Agent Standardization initiative (Epic #1079): converts the Playwright Testing Agent ChatGPT/Codex export into a standardised multi-provider agent (Claude, GitHub Copilot, OpenAI) and lands the supporting schema, hook, and instruction infrastructure that the remaining 15 agents will reuse.
Linked issues
Closes#1087
Relates to #1079, #1104, #1105, #1106
Changelog
Added
agents/playwright-testing-agent/): provider-agnosticAGENT.md+shared/core-prompt.mdfaithful to the real WordPress/WooCommerce test-pack-builder review-before-code workflow;claude/,copilot/,openai/configs + tool/skill definitions; agent-level.github/metadata; original export preserved in place. (Closesfeat(agents): rewrite Playwright Testing Agent for multi-provider support (Phase 1 PILOT) #1087)lightspeed-playwright-testingplugin with four provider manifests, registered inPLUGIN_MANIFEST.json. (Relates to Epic: Multi-Provider Agent Standardization Initiative #1079)multi-provider-agent,agent-plugin-binding,provider-config,agent-capability-manifest) with examples and registry entries. (Relates to automation(schema): add 4 multi-provider agent JSON schemas #1104)agent-spec-validator,multi-provider-consistency-checker,plugin-integrity-checker,agent-security-auditor) with unit tests and registry entries. (Relates to automation(hooks): add 4 agent/plugin validation hooks #1105)Summary
plugins/lightspeed-playwright-testing/with.claude-plugin,.codex-plugin,.gemini-plugin, andcopilot-plugin.json, registered in the plugin registry.Testing
markdownlint(23 new files) — 0 errorseslint(hooks) — cleanspectralYAML — no errorsvalidate:json,validate:json:schemas,validate:plugins,validate:skill-manifests,validate:retired-doc-links,validate:issue-fields— passchangelogvalidators — passjestsuite — 690/690 pass (includes 16 new hook tests)Notes for reviewers
agents/,plugins/,schema/,hooks/,instructions/,cookbook/folders per CLAUDE.md; audit/framework reports under.github/reports/.Checklist (Global DoD / PR)
🤖 Generated with Claude Code