fix: Make confidence-status assertions deterministic in triage tests by ashleyshaw · Pull Request #1712 · lightspeedwp/.github

ashleyshaw · 2026-08-10T16:31:45Z

Summary

Addresses CodeRabbit findings from PR #1703 by making confidence-status assertions deterministic in the handle-needs-triage test suite. These were functional correctness issues flagged during code review of the merged Phase 5 work.

Changes

Line 247: Assert result.status is specifically 'warning' for low-confidence, unlabeled fixture at 0.95 threshold
- Removed permissive assertion allowing both 'warning' and 'preview'
- Now strictly expects 'warning' when confidence is below threshold
Lines 294-296: Derive expected status from returned inference confidences
- Returns 'preview' if any inference meets confidence threshold (0.85)
- Returns 'warning' if no inference meets threshold
- Removed permissive assertion that allowed 'skipped' status

Test Results

✅ All 27 handle-needs-triage tests passing
✅ Linting and formatting automated fixes applied

Linked issues

Closes#1709
Relates to #1679

Changelog

Fixed

Make confidence-status assertions deterministic in handle-needs-triage tests
Removed overly permissive test expectations that allowed multiple status values
Tests now derive expected status from actual inference confidences

Checklist (Global DoD / PR)

Code changes complete and tested locally
All 27 tests passing
Linting and formatting passed
Deterministic test assertions
Documentation in code comments

coderabbitai · 2026-08-10T16:32:02Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 9ea83521-9d86-40f7-8528-e88b5da17edd

📥 Commits

Reviewing files that changed from the base of the PR and between 802ea55 and 96038e7.

📒 Files selected for processing (1)

scripts/automation/__tests__/handle-needs-triage.test.js

📝 Walkthrough

Summary by CodeRabbit

Tests
- Improved automated coverage for issue triage confidence thresholds.
- Added clearer validation for low-confidence outcomes, including warning states.
- Updated checks to consistently verify preview and warning results based on inferred confidence.
- These improvements help ensure triage decisions behave predictably across different confidence levels.

Walkthrough

The pull request updates two triage automation tests. The tests now assert warning for low confidence and derive preview or warning from inferred confidence.

Changes

Triage confidence tests

Layer / File(s)	Summary
Deterministic confidence assertions `scripts/automation/__tests__/handle-needs-triage.test.js`	The low-confidence test requires `warning` at a `0.95` threshold. The threshold test expects `preview` when type or area confidence reaches `0.85`; otherwise, it expects `warning`.

Estimated code review effort: 2 (Simple) | ~10 minutes

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

Create stacked PR
Commit on current branch

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/handle-needs-triage-confidence-assertions

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-08-10T16:32:15Z

⏱️ Aging and SLA annotation

Age: 0 day(s)
SLA state: Within SLA
Thresholds: warn at 7 days, breach at 14 days
Last updated: 2026-08-10T16:44:10.916Z

Maintained by project-meta-sync workflow.

github-actions · 2026-08-10T16:32:36Z

✅ Template check passed after update. Thanks for fixing the PR description.

github-actions · 2026-08-10T16:36:10Z

🔍 Reviewer Summary for PR #1712

CI Status: ✅ success
Files changed: 1
Risk Distribution: 0 critical, 0 high, 1 medium, 0 low

Recommendations

Ready to proceed pending human review

…triage tests Address CodeRabbit findings on PR #1703: 1. Assert result.status is specifically 'warning' for low-confidence, unlabeled fixture at 0.95 threshold (line 247) 2. Derive expected status from returned inference confidences instead of accepting all three statuses (lines 294-296) - Returns 'preview' if any inference meets 0.85 threshold - Returns 'warning' if no inference meets threshold All 27 handle-needs-triage tests passing. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

- OPENSPEC frontmatter/footer validation fixes (PR #1711) - Confidence-status assertion determinism in handle-needs-triage tests (PR #1712) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

Resolved multiple-footer validation errors by removing old footer text and keeping only the latest structured footer with document metadata. - badges-workflow-integration: v1.2.0 → v1.2.1 - github-actions-v7-upgrade: 1.1.1 (updated from 1.1) - issue-metadata-triage-expansion: 1.1.1 (updated from 1.1.0) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix: Correct frontmatter validation errors in project documentation Fixed invalid status values and missing required fields in OPENSPEC analyses: - Changed status from 'in-progress'/'implemented' to 'active' (valid enum) - Changed version from number to string format ("1.1") - Added missing description fields - Added owners field to all specification documents - Standardized frontmatter field formatting All frontmatter validation now passes (4/4 files validated, 0 errors). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: Add proper document footers to OPENSPEC files Added footer sections to all OPENSPEC analysis and specification documents with metadata and status information for maintainability and tracking. Resolves footer validation errors in CI/CD checks. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: Bump OPENSPEC document versions to reflect footer additions Updated version numbers to satisfy frontmatter freshness validation: - badges-workflow-integration: v1.2.0 → v1.2.1 - github-actions-v7-upgrade: 1.1 → 1.1.1 - issue-metadata-triage-expansion: 1.1.0 → 1.1.1 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * docs: Add changelog entries for PR #1711 and PR #1712 - OPENSPEC frontmatter/footer validation fixes (PR #1711) - Confidence-status assertion determinism in handle-needs-triage tests (PR #1712) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * docs: Update CHANGELOG.md last_updated to 2026-08-10 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: Remove duplicate footer sections from OPENSPEC files Resolved multiple-footer validation errors by removing old footer text and keeping only the latest structured footer with document metadata. - badges-workflow-integration: v1.2.0 → v1.2.1 - github-actions-v7-upgrade: 1.1.1 (updated from 1.1) - issue-metadata-triage-expansion: 1.1.1 (updated from 1.1.0) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>

…ions

ashleyshaw force-pushed the fix/handle-needs-triage-confidence-assertions branch from 48bbfca to 1143387Compare August 10, 2026 16:38

ashleyshaw removed the meta:needs-changelog Requires a changelog entry before merge label Aug 10, 2026

ashleyshawand others added 2 commits August 10, 2026 18:46

docs: Add changelog entries for PR#1711and PR#1712
87db8a2
- OPENSPEC frontmatter/footer validation fixes (PR #1711) - Confidence-status assertion determinism in handle-needs-triage tests (PR #1712) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

docs: Update CHANGELOG.md last_updated to 2026-08-10
d6a9380
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

github-actionsBot added area:documentation Docs & guides lang:md Markdown content/docs meta:needs-changelog Requires a changelog entry before merge labels Aug 10, 2026

ashleyshaw enabled auto-merge (squash) August 10, 2026 17:14

Merge branch 'develop' into fix/handle-needs-triage-confidence-assert…
96038e7
…ions

ashleyshaw merged commit 30cd434 into developAug 10, 2026
19 of 23 checks passed

ashleyshaw deleted the fix/handle-needs-triage-confidence-assertions branch August 10, 2026 17:15

Navigation Menu

Uh oh!

fix: Make confidence-status assertions deterministic in triage tests - #1712

fix: Make confidence-status assertions deterministic in triage tests#1712
ashleyshaw merged 5 commits into
developfrom
fix/handle-needs-triage-confidence-assertions

ashleyshaw commented Aug 10, 2026•
edited
Loading

Uh oh!

coderabbitaiBot commented Aug 10, 2026•
edited
Loading

Review failed

Summary by CodeRabbit

Walkthrough

Changes

Uh oh!

github-actionsBot commented Aug 10, 2026•
edited
Loading

Uh oh!

github-actionsBot commented Aug 10, 2026•
edited
Loading

Uh oh!

github-actionsBot commented Aug 10, 2026•
edited
Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ashleyshaw commented Aug 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test Results

Linked issues

Changelog

Fixed

Checklist (Global DoD / PR)

Uh oh!

coderabbitaiBot commented Aug 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Summary by CodeRabbit

Walkthrough

Changes

Uh oh!

github-actionsBot commented Aug 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⏱️ Aging and SLA annotation

Uh oh!

github-actionsBot commented Aug 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Aug 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Reviewer Summary for PR #1712

Recommendations

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ashleyshaw commented Aug 10, 2026•
edited
Loading

coderabbitaiBot commented Aug 10, 2026•
edited
Loading

github-actionsBot commented Aug 10, 2026•
edited
Loading

github-actionsBot commented Aug 10, 2026•
edited
Loading

github-actionsBot commented Aug 10, 2026•
edited
Loading