Skip to content

design: Testing agent multi-framework architecture — Phase planning - #1797

Merged
ashleyshaw merged 2 commits into
developfrom
design/testing-agent-multi-framework-architecture
Aug 12, 2026
Merged

design: Testing agent multi-framework architecture — Phase planning#1797
ashleyshaw merged 2 commits into
developfrom
design/testing-agent-multi-framework-architecture

Conversation

@ashleyshaw

@ashleyshawashleyshaw commented Aug 12, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Testing Agent Multi-Framework Architecture — Phase Planning

This repository enforces changelog, release, and label automation for all PRs and issues.
See the organisation-wide Automation Governance & Release Strategy for contributor rules.

Linked issues

Resolves#1799

Changelog

Added

  • ARCHITECTURE_DIAGRAMS.md — 8 Mermaid diagrams for testing infrastructure (2-tier delegation, framework matrix, CI/CD integration)
  • QUESTIONS_AND_ANSWERS.md — 5 scope decisions with rationale (frameworks, coverage thresholds, documentation strategy)
  • IMPLEMENTATION_PLAN.md — 4-phase roadmap with 54-79 hour effort estimate (Phase 1-4 task breakdown)
  • TESTING_AGENT_REWRITE_PROMPT.md — Updated rewrite guidance for .github/agents/testing.agent.md

Changed

  • Branch renamed from design/testing-agent-multi-framework-architecture to design/testing-agent-expansion (improved naming)

Summary

Planning documentation for consolidating LightSpeed's testing infrastructure into a unified, multi-framework testing agent supporting Jest, PHPUnit, pytest, and Playwright.

Key recommendation: Rename and expand agents/playwright-testing-agent/ → agents/testing-agent/ to serve all org-wide testing needs, while .github/agents/testing.agent.md becomes a lightweight control-plane coordinator.

Deliverables

📋 Documents

  • README.md — Project overview with 5 key scope questions to resolve
  • QUESTIONS_AND_ANSWERS.md — Answers with best practices and rationale
  • IMPLEMENTATION_PLAN.md — Detailed task breakdown for Phases 1-4 (54-79 hours estimated)
  • ARCHITECTURE_DIAGRAMS.md — 8 Mermaid diagrams showing coordination patterns
  • TESTING_AGENT_REWRITE_PROMPT.md — Updated rewrite guidance

✅ Key Recommendations

QuestionRecommendation
ScopeAll LightSpeed repositories
FrameworksJest, PHPUnit, pytest, Playwright (all)
CoverageDifferentiated thresholds (85% plugins, 80% themes, 75% control-plane)
DocumentationComprehensive (guides, skills, provider configs)
TestingUnit + Integration + E2E validation

Architecture Overview

Control-Plane (.github) ↓ delegates to ↓
Portable Agent (agents/testing-agent/)
├─ Jest Provider (JavaScript/TypeScript)
├─ PHPUnit Provider (PHP)
├─ pytest Provider (Python)
└─ Playwright Provider (Browser E2E)

Implementation Timeline

  • Phase 1: Design & Planning (6-14 hours) — Ready to start
  • Phase 2: Portable agent expansion (27-35 hours) — Jest, PHPUnit, pytest skills
  • Phase 3: Control-plane agent rewrite (5-7 hours) — Coordination logic
  • Phase 4: Testing & documentation (16-23 hours) — Full validation

Total estimated effort: 54-79 hours (~1.5-2 weeks)

Checklist (Global DoD / PR)

  • All AC met and demonstrated
  • Accessibility checklist completed (where relevant):
    • Semantic HTML and heading order verified
    • Contrast and non-colour cues reviewed (WCAG 2.2 AA or higher)
  • Docs/readme/changelog updated (if user-facing)
  • Code/design reviews (planning documents ready for team approval)
  • CI validation pending (branch rename and template fixes applied)

🧱 Built by LightSpeedWP with ☕, 🚀, and open-source spirit!

Co-Authored-By: Claude Haiku 4.5 noreply@anthropic.com

@ashleyshaw@claude
design: Phase planning for testing agent multi-framework architecture
e95baa4
Create active project for consolidating testing infrastructure into unified
multi-framework agent. Includes:
- README.md: Project overview and key questions to resolve
- QUESTIONS_AND_ANSWERS.md: Scope recommendations with best practices
- IMPLEMENTATION_PLAN.md: Detailed task breakdown for Phases 1-4
- ARCHITECTURE_DIAGRAMS.md: 8 Mermaid diagrams showing:
* 2-tier agent architecture (control-plane + portable)
* Delegation flow and decision logic
* Test execution pipeline
* Framework coverage matrix
* Skill and documentation structure
* Integration test scenarios
* Coverage thresholds by context
* CI/CD integration flow
- TESTING_AGENT_REWRITE_PROMPT.md: Rewrite guidance (updated)
Plan consolidates agents/playwright-testing-agent → agents/testing-agent
with Jest, PHPUnit, pytest support. Control-plane coordinator delegates
to portable agent for actual test execution.
Total estimated effort: 54-79 hours (~1.5-2 weeks)
Branch: design/testing-agent-multi-framework-architecture
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@coderabbitai

coderabbitaiBot commented Aug 12, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@ashleyshaw, you've reached your PR review limit, so we couldn't start this review.

Next review available in:22 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 47b7dab6-576b-439f-9bc2-f78999458c9f

📥 Commits

Reviewing files that changed from the base of the PR and between 752f6e1 and b2e1001.

📒 Files selected for processing (5)
  • .github/projects/active/testing-agent-multi-framework-2026-08-12/ARCHITECTURE_DIAGRAMS.md
  • .github/projects/active/testing-agent-multi-framework-2026-08-12/IMPLEMENTATION_PLAN.md
  • .github/projects/active/testing-agent-multi-framework-2026-08-12/QUESTIONS_AND_ANSWERS.md
  • .github/projects/active/testing-agent-multi-framework-2026-08-12/README.md
  • .github/projects/active/testing-agent-multi-framework-2026-08-12/TESTING_AGENT_REWRITE_PROMPT.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actionsBot commented Aug 12, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

✅ Template check passed after update. Thanks for fixing the PR description.

@github-actionsgithub-actionsBot added the status:needs-review Awaiting code review label Aug 12, 2026
@github-actions

github-actionsBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

❌ Branch Name Validation Failed

The branch name design/testing-agent-multi-framework-architecture does not follow the LightSpeed branching strategy.

Required Format

{type}/{scope}-{short-title}
  • type: one of the allowed prefixes (lowercase)
  • scope: lowercase, hyphens only (no underscores or uppercase)
  • title: lowercase, hyphens only (no underscores or uppercase)

Allowed Branch Types

feat, fix, hotfix, release, refactor, chore, docs, test, perf, ci, build, deps, security, revert, research, design, a11y, ux, i18n, ops, proto, ds, api, schema, telemetry, content, seo, config, migrate, qa, uat, audit, codex

Valid Examples

  • ✓ feat/branch-naming-enforcement
  • ✓ fix/validation-script-bug
  • ✓ chore/update-dependencies
  • ✓ docs/branching-strategy-guide
  • ✓ hotfix/critical-security-patch

Invalid Examples

  • ✗ claude/my-branch (type "claude" not allowed)
  • ✗ Feature/MyBranch (uppercase not allowed)
  • ✗ fix-bug (missing type prefix)
  • ✗ feat/my_feature (underscores not allowed)
  • ✗ feat/MyFeature (uppercase not allowed)

Solution

Rename your branch to follow the pattern and update the PR.

For more information, see docs/BRANCHING_STRATEGY.md.

@github-actions

github-actionsBot commented Aug 12, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

⏱️ Aging and SLA annotation

  • Age: 0 day(s)
  • SLA state: Within SLA
  • Thresholds: warn at 7 days, breach at 14 days
  • Last updated: 2026-08-12T09:49:28.270Z

Maintained by project-meta-sync workflow.

@github-actionsgithub-actionsBot added type:design Design work priority:normal Default priority type:chore Chore / small hygiene change type:test Testing/coverage meta:needs-changelog Requires a changelog entry before merge labels Aug 12, 2026
@github-actions

github-actionsBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

📄 README Validation

✅ All README checks passed.

CheckResult
✅ FrontmatterPassed
✅ StructurePassed

@github-actions

github-actionsBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

🎨 Mermaid Diagram Validation

❌ One or more Mermaid diagram checks failed.

CheckResult
✅ SyntaxPassed
✅ AccessibilityPassed
❌ Colour ContrastFailed

@github-actions

github-actionsBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

🔗 Project Linking Validation

Projects Checked: 36
Projects with Links: 32

❌ Missing Related Issues Section

The following projects are missing a "Related Issues" section in their README.md:

  • release-agentic-workflows-2026-08-11

See Linking Standard for format.

Validation Date: 2026-08-12T09:12:38.643Z
Validator: GitHub Actions

@ashleyshawashleyshaw mentioned this pull request Aug 12, 2026
Closed
29 tasks
@github-actions

github-actionsBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

🔍 Reviewer Summary for PR #1797

CI Status: ✅ success
Files changed: 5
Risk Distribution: 0 critical, 5 high, 0 medium, 0 low

Recommendations

  • Ready to proceed pending human review

@github-actionsgithub-actionsBot removed type:design Design work type:chore Chore / small hygiene change labels Aug 12, 2026
@ashleyshaw
ashleyshaw enabled auto-merge (squash) August 12, 2026 09:44
@ashleyshawashleyshaw self-assigned this Aug 12, 2026
@github-actionsgithub-actionsBot added the type:design Design work label Aug 12, 2026
@ashleyshaw
ashleyshaw merged commit 5dab643 into developAug 12, 2026
22 of 28 checks passed
@ashleyshaw
ashleyshaw deleted the design/testing-agent-multi-framework-architecture branch August 12, 2026 09:52
@github-actions

github-actionsBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

🔗 Project Linking Validation

Projects Checked: 38
Projects with Links: 34

❌ Missing Related Issues Section

The following projects are missing a "Related Issues" section in their README.md:

  • release-agentic-workflows-2026-08-11

See Linking Standard for format.

Validation Date: 2026-08-12T09:54:11.007Z
Validator: GitHub Actions

ashleyshaw added a commit that referenced this pull request Aug 12, 2026
@ashleyshaw@claude
openspec: Testing Agent Multi-Framework Architecture specification
8bd1316
- Proposal: Consolidate testing frameworks into portable agent
- Design: 2-tier architecture with coordination protocol
- Tasks: Phase 2-4 implementation breakdown (54-79 hours)
- Framework support: Jest, PHPUnit, pytest, Playwright
- WordPress integration: Standards for all frameworks
- Coverage thresholds: Differentiated by project type
Relates to: #1799#1797
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This was referenced Aug 12, 2026
Closed
Closed
ashleyshaw added a commit that referenced this pull request Aug 12, 2026
@ashleyshaw@claude
docs: Update project documentation with GitHub issue references
7156ca1
- Add GitHub issue/PR table to active project README
- Link master issue #1799, planning PR #1797, OpenSpec issue #1825, OpenSpec PR #1824
- Update OpenSpec README with related issues section
- Cross-reference all coordination points
Relates to: #1799#1797#1825#1824
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
ashleyshaw added a commit that referenced this pull request Aug 12, 2026
@ashleyshaw@claude
openspec: Testing Agent Multi-Framework Architecture specification
459ecff
- Proposal: Consolidate testing frameworks into portable agent
- Design: 2-tier architecture with coordination protocol
- Tasks: Phase 2-4 implementation breakdown (54-79 hours)
- Framework support: Jest, PHPUnit, pytest, Playwright
- WordPress integration: Standards for all frameworks
- Coverage thresholds: Differentiated by project type
Relates to: #1799#1797
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
ashleyshaw added a commit that referenced this pull request Aug 12, 2026
@ashleyshaw@claude
docs: Update project documentation with GitHub issue references
be1a989
- Add GitHub issue/PR table to active project README
- Link master issue #1799, planning PR #1797, OpenSpec issue #1825, OpenSpec PR #1824
- Update OpenSpec README with related issues section
- Cross-reference all coordination points
Relates to: #1799#1797#1825#1824
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
ashleyshaw added a commit that referenced this pull request Aug 12, 2026
@ashleyshaw@claude
design: Phase planning for testing agent multi-framework architecture (…
cbc4728
…#1797)
Create active project for consolidating testing infrastructure into unified
multi-framework agent. Includes:
- README.md: Project overview and key questions to resolve
- QUESTIONS_AND_ANSWERS.md: Scope recommendations with best practices
- IMPLEMENTATION_PLAN.md: Detailed task breakdown for Phases 1-4
- ARCHITECTURE_DIAGRAMS.md: 8 Mermaid diagrams showing:
* 2-tier agent architecture (control-plane + portable)
* Delegation flow and decision logic
* Test execution pipeline
* Framework coverage matrix
* Skill and documentation structure
* Integration test scenarios
* Coverage thresholds by context
* CI/CD integration flow
- TESTING_AGENT_REWRITE_PROMPT.md: Rewrite guidance (updated)
Plan consolidates agents/playwright-testing-agent → agents/testing-agent
with Jest, PHPUnit, pytest support. Control-plane coordinator delegates
to portable agent for actual test execution.
Total estimated effort: 54-79 hours (~1.5-2 weeks)
Branch: design/testing-agent-multi-framework-architecture
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
@ashleyshawashleyshaw mentioned this pull request Aug 12, 2026
Merged
17 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@ashleyshawashleyshaw

Labels

meta:needs-changelogRequires a changelog entry before mergepriority:normalDefault prioritystatus:needs-reviewAwaiting code reviewtype:designDesign worktype:testTesting/coverage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Testing Agent Multi-Framework Architecture — Phase Planning

1 participant

@ashleyshaw