Skip to content

fix: Fix changelog regex bugs and strengthen test assertions - #1729

Merged
ashleyshaw merged 4 commits into
developfrom
fix/changelog-regex-and-test-assertions
Aug 11, 2026
Merged

fix: Fix changelog regex bugs and strengthen test assertions#1729
ashleyshaw merged 4 commits into
developfrom
fix/changelog-regex-and-test-assertions

Conversation

@ashleyshaw

@ashleyshawashleyshaw commented Aug 11, 2026

Copy link
Copy Markdown
Member

Summary

Fixed 3 changelog regex validation bugs and strengthened test assertions to improve code quality and reliability.

Issue #1715 - Changelog Validation Regex Bugs (3 fixes):

  1. Line 31-40 (changelogValidator.cjs): Fixed em-dash validation regex that incorrectly flagged all hyphenated words. Now only flags spaced hyphens (-) used as pauses, not compound words like "backwards-compatible".
  2. Line 174 (keepAChangelogParser.cjs): Replaced Perl syntax \z with JavaScript anchor $ for end-of-string validation.
  3. Line 186 (keepAChangelogParser.cjs): Added regex metacharacter escaping for category names to handle special chars.

Issue #1716 - Strengthen Test Assertions:

  • Replaced overly-permissive conditional test assertions with deterministic behavior verification
  • Added 7 new edge case tests for empty content, whitespace, boundary conditions
  • Documented and tested 0.85 confidence threshold multiplier
  • All 38 tests passing in handle-needs-triage.test.js

Linked Issues

Closes#1715 - Changelog regex validation bugs
Closes#1716 - Handle-needs-triage test assertion improvements
Related to #1714 - gitOps.cjs cross-repo isolation work

Changelog

Checklist (Global DoD / PR)

  • Changes are isolated and focused on the stated issues
  • All new code includes tests
  • Test coverage maintained/improved
  • All tests passing (89.06% for handle-needs-triage.js)
  • Commit messages reference issue numbers
  • PR description includes linked issues
  • PR description includes changelog
  • Code follows repository standards
  • No breaking changes introduced

🤖 Generated with Claude Code

@github-actions

github-actionsBot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

✅ Template check passed after update. Thanks for fixing the PR description.

@github-actions

github-actionsBot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

⏱️ Aging and SLA annotation

  • Age: 0 day(s)
  • SLA state: Within SLA
  • Thresholds: warn at 7 days, breach at 14 days
  • Last updated: 2026-08-11T09:33:49.188Z

Maintained by project-meta-sync workflow.

@github-actionsgithub-actionsBot added status:needs-review Awaiting code review type:bug Bug or defect priority:normal Default priority area:documentation Docs & guides area:tests Test suites & harnesses area:scripts Scripts & tooling labels Aug 11, 2026
@coderabbitai

coderabbitaiBot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@ashleyshaw, you've reached your PR review limit, so we couldn't start this review.

Next review available in:41 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 9b76a00d-460c-4f46-bac0-57855bbdd0e9

📥 Commits

Reviewing files that changed from the base of the PR and between 96aad40 and 4f15688.

📒 Files selected for processing (1)
  • CHANGELOG.md
📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes

    • Improved changelog validation for spaced hyphens used as em dashes while preserving valid compound words.
    • Fixed changelog parsing for end-of-input handling and category names containing special characters.
  • Tests

    • Added comprehensive coverage for changelog parsing, validation, release handling, Git operations, and triage recommendations.
    • Strengthened confidence, threshold, boundary, and deterministic behaviour checks.
  • Documentation

    • Added an Unreleased changelog entry documenting these improvements.

Walkthrough

The changes fix changelog validation and parsing regexes. They add parser and validator edge-case tests, Git operations tests, and precise triage confidence and threshold assertions. The changelog records these updates.

Changes

Changelog validation and parsing

Layer / File(s)Summary
Changelog validation rules
agents/changelog/includes/changelogValidator.cjs, agents/changelog/includes/tests/changelogValidator.test.cjs
Validation now rejects spaced hyphens as em-dashes while accepting hyphenated compound words.
Changelog parser regexes and coverage
agents/changelog/includes/keepAChangelogParser.cjs, agents/changelog/includes/tests/keepAChangelogParser.test.cjs, CHANGELOG.md
Parser regexes use JavaScript end anchors and escaped category names. Tests cover parsing, appending, release conversion, excerpts, version listing, and version checks. The changelog records the fixes and coverage updates.

Git operations test coverage

Layer / File(s)Summary
Git operations behaviour tests
agents/release/__tests__/gitOps.test.js
Tests cover directory validation, sanitised errors, repository state, required methods, null handling, and exports.

Triage inference test coverage

Layer / File(s)Summary
Inference and threshold assertions
scripts/automation/__tests__/handle-needs-triage.test.js
Tests now assert exact confidence values, threshold boundaries, deterministic results, area filtering, multiple matches, empty input, and CI-based assignment.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

Suggested labels:area:automation

Suggested reviewers:krugazul

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check nameStatusExplanationResolution
Out of Scope Changes check⚠️ WarningThe new gitOps tests and handle-needs-triage assertion changes are unrelated to the linked changelog regex issue [#1715].Move the gitOps and handle-needs-triage changes to separate PRs, or link issues that explicitly require them.
Description check⚠️ WarningThe description covers the summary, linked issues, changelog, and checklist, but omits the required risk assessment and testing instructions.Add the Risk Assessment and How to Test sections, including risk, impact, mitigation, prerequisites, test steps, expected results, and edge cases.
✅ Passed checks (3 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedThe PR fixes all three regex defects and adds coverage for hyphen boundaries, JavaScript anchors, special-character categories, and changelog regressions [#1715].
Docstring Coverage✅ PassedNo functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check✅ PassedThe title clearly identifies both primary changes: changelog regex fixes and stronger test assertions.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/changelog-regex-and-test-assertions

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actionsgithub-actionsBot added lang:js JavaScript/TypeScript lang:md Markdown content/docs type:chore Chore / small hygiene change meta:needs-changelog Requires a changelog entry before merge labels Aug 11, 2026
@coderabbitai
coderabbitaiBot requested a review from krugazulAugust 11, 2026 08:09
Comment threadagents/release/__tests__/gitOps.test.js Fixed
@coderabbitaicoderabbitaiBot added the area:automation Automation workflows and agents label Aug 11, 2026

@coderabbitaicoderabbitaiBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@agents/changelog/includes/tests/keepAChangelogParser.test.cjs`:
- Around line 1-5: Add the required test-file header at the top of
keepAChangelogParser.test.cjs, before the imports, documenting the test purpose,
author, date, and related files. Keep the existing imports and test
implementation unchanged.
- Line 7: Update keepAChangelogParser.test.cjs by adding the required purpose,
author, date, and related-files header, and make __dirname available for this
CommonJS test by declaring it in the applicable eslint.config.cjs globals for
**/*.cjs or locally in the test file.
In `@agents/release/__tests__/gitOps.test.js`:
- Around line 18-22: Update the test for validateDirectory to capture the thrown
error and assert its message does not contain the supplied absolute directory
path. Then update validateDirectory so its error message is sanitized and
excludes user-provided directory values while preserving the invalid-directory
failure behavior.
- Around line 1-4: Update the header comment in gitOps.test.js to include the
required author, date, and related-files fields while preserving its existing
purpose description. Use the project’s standard test-header format and identify
the relevant files covered by these gitOps tests.
- Around line 116-120: Align both affected tests in
agents/release/__tests__/gitOps.test.js:116-120 and
agents/release/__tests__/gitOps.test.js:138-145 with the actual exports from
agents/release/includes/gitOps.cjs. Prefer replacing the GitRepository,
sanitizeErrorMessage, and validateDirectory tests with tests of the exported
function-based API; only implement and export the class, add the test file to
npm test, and retain the workDir contract assertion if the class API is
intentionally required.
In `@CHANGELOG.md`:
- Line 40: Update the changelog entry’s “behavior” spelling to “behaviour,”
preserving the surrounding wording and formatting.
In `@scripts/automation/__tests__/handle-needs-triage.test.js`:
- Around line 449-455: Validate confidenceThreshold at the handler’s
input/configuration boundary, accepting only values from 0 through 1 and
returning the established invalid-configuration result for out-of-range values
such as 2.0. Update the test around handler.processIssue to assert that invalid
configuration outcome instead of a warning, while preserving valid-threshold
behavior.
- Around line 65-73: Strengthen the deterministic selection assertions in
scripts/automation/__tests__/handle-needs-triage.test.js at lines 65-73, 76-85,
and 458-473: call inferType twice at lines 65-73 and compare both selected type
and confidence; at lines 76-85 derive the maximum entry from result.scores and
assert the returned type and confidence match it; at lines 458-473 assert
processIssue’s inferred type and confidence match its highest score. Include any
required lint fixes and a brief rationale summarizing the test changes.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: c9403c92-4fe1-4ec7-94ce-28ccafeaa44a

📥 Commits

Reviewing files that changed from the base of the PR and between 1417d8e and 96aad40.

📒 Files selected for processing (7)
  • CHANGELOG.md
  • agents/changelog/includes/changelogValidator.cjs
  • agents/changelog/includes/keepAChangelogParser.cjs
  • agents/changelog/includes/tests/changelogValidator.test.cjs
  • agents/changelog/includes/tests/keepAChangelogParser.test.cjs
  • agents/release/__tests__/gitOps.test.js
  • scripts/automation/__tests__/handle-needs-triage.test.js
📜 Review details
⏰ Context from checks skipped due to timeout. (4)
  • GitHub Check: Testing
  • GitHub Check: coderabbit-gate
  • GitHub Check: Analyze (python)
  • GitHub Check: Summary
⚠️ CI failures not shown inline (2)

GitHub Actions: Validate PR Template / 0_validate-pr-template.txt: fix: Fix changelog regex bugs and strengthen test assertions

Conclusion: failure

View job details

##[group]Run actions/github-script@v7
with:
script: const { validatePullRequestBody } = require('./scripts/validation/template-helpers.cjs');
const marker = '<!-- template-enforcement -->';
const pr = context.payload.pull_request;
const author = pr.user?.login || '';
const isDependabot = author === 'dependabot[bot]' || author === 'app/dependabot';
const isImgbot = author === 'imgbot[bot]' || author === 'app/imgbot';
if (isDependabot || isImgbot) {
core.info(`Skipping PR template validation for bot author ${author}.`);
return;
}
const validation = validatePullRequestBody(pr.body || '', pr.labels || [], pr.head?.ref || '');
const comments = await github.paginate(github.rest.issues.listComments, {
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: pr.number,
per_page: 100
});
const previous = comments.find((comment) =>
comment.user?.type === 'Bot' && comment.body?.includes(marker)
);
if (validation.missing.length === 0) {
if (previous) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: previous.id,
body: `${marker}\n✅ Template check passed after update. Thanks for fixing the PR description.`
});
}
return;
}
const message = [
marker,
'🚫 This PR description is missing required template content.',
'',
`Missing required section(s): ${validation.missing.join(', ')}`,
'',
'Please update the PR body using one of the repository PR templates:',
'- https://github.com/lightspeedwp/.github/blob/develop/.github/pull_request_template.md',
'- https://github.com/lightspeedwp/.github/tree/develop/.github/PULL_REQUEST_TEMPLATE',
'',
'Empty placeholders, unchecked checklist boxes, and stub issue references do not count.'
].join('\n');
if (previous) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: previous.id,
body: message
});
} else {
await github.rest.issues....

GitHub Actions: Validate PR Template / validate-pr-template: fix: Fix changelog regex bugs and strengthen test assertions

Conclusion: failure

View job details

##[group]Run actions/github-script@v7
with:
script: const { validatePullRequestBody } = require('./scripts/validation/template-helpers.cjs');
const marker = '<!-- template-enforcement -->';
const pr = context.payload.pull_request;
const author = pr.user?.login || '';
const isDependabot = author === 'dependabot[bot]' || author === 'app/dependabot';
const isImgbot = author === 'imgbot[bot]' || author === 'app/imgbot';
if (isDependabot || isImgbot) {
core.info(`Skipping PR template validation for bot author ${author}.`);
return;
}
const validation = validatePullRequestBody(pr.body || '', pr.labels || [], pr.head?.ref || '');
const comments = await github.paginate(github.rest.issues.listComments, {
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: pr.number,
per_page: 100
});
const previous = comments.find((comment) =>
comment.user?.type === 'Bot' && comment.body?.includes(marker)
);
if (validation.missing.length === 0) {
if (previous) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: previous.id,
body: `${marker}\n✅ Template check passed after update. Thanks for fixing the PR description.`
});
}
return;
}
const message = [
marker,
'🚫 This PR description is missing required template content.',
'',
`Missing required section(s): ${validation.missing.join(', ')}`,
'',
'Please update the PR body using one of the repository PR templates:',
'- https://github.com/lightspeedwp/.github/blob/develop/.github/pull_request_template.md',
'- https://github.com/lightspeedwp/.github/tree/develop/.github/PULL_REQUEST_TEMPLATE',
'',
'Empty placeholders, unchecked checklist boxes, and stub issue references do not count.'
].join('\n');
if (previous) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: previous.id,
body: message
});
} else {
await github.rest.issues....
🧰 Additional context used
📓 Path-based instructions (9)
**/*

📄 CodeRabbit inference engine (CLAUDE.md)

**/*: Do not place reusable assets under .github/; use the matching top-level portable folder instead.
Use UK English throughout, including spellings such as optimise, organisation, colour, and behaviour.
Validate all input, escape all output, use nonces, and never commit secrets.
Do not move existing agents, instructions, or schemas without a migration issue recording the source path, target path, and validation plan.
Do not add WordPress plugin- or theme-specific project code to the .github control plane.
Do not commit node_modules/, build/, or other generated artefacts.

**/*: All code changes must include lint fixes, relevant tests, and a short rationale summarising the change.
Never output secrets; treat production and customer data as sensitive; follow the OWASP Top 10 for web security.
Every agent must follow the applicable AGENT_STANDARDS.md template, and contributors must follow the organisation-wide coding standards.
Before editing, validate the branch with npm run validate:branch-name -- --branch <name>; use {type}/{scope}-{short-title}, target develop except for release/hotfix branches targeting main, never use a claude/ prefix, and delete merged branches.
Prefer minimal, modular solutions; justify heavier tools by their return on investment and maintenance cost.
When requirements are uncertain, propose safe defaults and ask one focused clarification question.

Files:

  • agents/release/__tests__/gitOps.test.js
  • agents/changelog/includes/tests/changelogValidator.test.cjs
  • scripts/automation/__tests__/handle-needs-triage.test.js
  • CHANGELOG.md
  • agents/changelog/includes/changelogValidator.cjs
  • agents/changelog/includes/tests/keepAChangelogParser.test.cjs
  • agents/changelog/includes/keepAChangelogParser.cjs
**/*.{js,jsx,ts,tsx}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{js,jsx,ts,tsx}: Use ESLint and Prettier for JavaScript and TypeScript code.
Avoid unnecessary JavaScript and defer or lazy-load it where possible; prefer native blocks.

Files:

  • agents/release/__tests__/gitOps.test.js
  • scripts/automation/__tests__/handle-needs-triage.test.js
**/agents/**/*

📄 CodeRabbit inference engine (CLAUDE.md)

Portable multi-file agent implementations belong in the root agents/ directory and must not assume .github/ paths.

Files:

  • agents/release/__tests__/gitOps.test.js
  • agents/changelog/includes/tests/changelogValidator.test.cjs
  • agents/changelog/includes/changelogValidator.cjs
  • agents/changelog/includes/tests/keepAChangelogParser.test.cjs
  • agents/changelog/includes/keepAChangelogParser.cjs
**/*.{php,js,jsx,ts,tsx,css,scss,html}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{php,js,jsx,ts,tsx,css,scss,html}: Follow WordPress Coding Standards and inline-documentation standards for PHP, JavaScript, CSS, and HTML.
Identify accessibility and performance issues during code review.

Files:

  • agents/release/__tests__/gitOps.test.js
  • scripts/automation/__tests__/handle-needs-triage.test.js
agents/**/*

📄 CodeRabbit inference engine (AGENTS.md)

Portable, reusable multi-file agents belong under agents/{name}-agent/ and must include AGENT.md plus provider-specific subdirectories where applicable.

Files:

  • agents/release/__tests__/gitOps.test.js
  • agents/changelog/includes/tests/changelogValidator.test.cjs
  • agents/changelog/includes/changelogValidator.cjs
  • agents/changelog/includes/tests/keepAChangelogParser.test.cjs
  • agents/changelog/includes/keepAChangelogParser.cjs
**/*.{js,ts}

⚙️ CodeRabbit configuration file

**/*.{js,ts}: Review JavaScript/TypeScript:

  • Ensure code is linted and follows project style guides.
  • Check for dead code, unused variables, and clear function naming.
  • Validate accessibility and performance optimisations.
  • Ensure tests are isolated and do not depend on external state.
  • Check for descriptive test names and clear test structure.

Files:

  • agents/release/__tests__/gitOps.test.js
  • scripts/automation/__tests__/handle-needs-triage.test.js
**/tests/*.*

⚙️ CodeRabbit configuration file

**/tests/*.*: Review all test files:

  • All test files must have a header (purpose, author, date, related files).
  • Use clear, descriptive test names and logical structure.
  • Include both positive and negative test cases.
  • Be discoverable from the main agent/test index.
  • Pass all style checks and linting.

Files:

  • agents/changelog/includes/tests/changelogValidator.test.cjs
  • agents/changelog/includes/tests/keepAChangelogParser.test.cjs
**/*.{md,mdx}

📄 CodeRabbit inference engine (CLAUDE.md)

Do not use a references frontmatter field; use inline links or footer sections instead.

Use UK English and optimise documentation and code explanations for clarity, scalability, maintainability, and profitable outcomes.

Files:

  • CHANGELOG.md
CHANGELOG.md

⚙️ CodeRabbit configuration file

CHANGELOG.md: Review CHANGELOG.md:

  • Confirm entries follow Keep a Changelog 1.1.0 format.
  • Each entry under [Unreleased] must include a PR link and issue link.
  • Verify entries use the correct section headings (Added, Changed, Fixed, Deprecated, Removed, Security, Documentation, Performance).
  • Check UK English spelling throughout.

Files:

  • CHANGELOG.md
🪛 ast-grep (0.45.1)
agents/release/__tests__/gitOps.test.js

[warning] 105-105: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(tempFile, "test")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').

(detect-non-literal-fs-filename)

🪛 ESLint
agents/changelog/includes/tests/keepAChangelogParser.test.cjs

[error] 7-7: '__dirname' is not defined.

(no-undef)

🪛 GitHub Actions: Meta Agent / 2_front-matter-validate.txt
CHANGELOG.md

[error] 1-1: Frontmatter freshness validation failed: the document body changed, but the 'last_updated' field was not updated (currently 2026-08-10). Command 'npm run validate:frontmatter:changed' failed with exit code 1.

🪛 GitHub Actions: Meta Agent / front-matter-validate
CHANGELOG.md

[error] 1-1: Frontmatter freshness validation failed: the body changed but the last_updated field was not updated (currently 2026-08-10). Command 'npm run validate:frontmatter:changed -- --base ... --head ...' failed with exit code 1.

🪛 GitHub Check: Linting
agents/release/__tests__/gitOps.test.js

[warning] 140-140:
'originalDir' is assigned a value but never used. Allowed unused vars must match /^_/u

🔇 Additional comments (3)
agents/changelog/includes/changelogValidator.cjs (1)

31-35: LGTM!

agents/changelog/includes/tests/changelogValidator.test.cjs (1)

97-109: LGTM!

Also applies to: 111-121, 123-134

agents/changelog/includes/keepAChangelogParser.cjs (1)

174-174: LGTM!

Also applies to: 186-187

Comment on lines +1 to +4
/**
* Tests for gitOps.cjs
* Tests for both function-based and class-based API
*/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add the required test header fields.

The header has a purpose, but it omits the author, date, and related files. Add these fields.

As per path instructions, test files must have a header with purpose, author, date, and related files.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@agents/release/__tests__/gitOps.test.js` around lines 1 - 4, Update the
header comment in gitOps.test.js to include the required author, date, and
related-files fields while preserving its existing purpose description. Use the
project’s standard test-header format and identify the relevant files covered by
these gitOps tests.

Source: Path instructions

Comment on lines +18 to +22
it("should validate directory exists", () => {
expect(() => {
validateDirectory("/nonexistent/path/that/does/not/exist");
}).toThrow();
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Test redaction for invalid-directory errors.

validateDirectory includes the supplied directory value in its thrown error. This assertion only checks that an error occurs, so it passes while an absolute work path remains exposed. Assert that the error does not contain the supplied path, then sanitise the implementation error before it is returned.

As per coding guidelines, validate all input, escape all output, use nonces, and never commit secrets.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@agents/release/__tests__/gitOps.test.js` around lines 18 - 22, Update the
test for validateDirectory to capture the thrown error and assert its message
does not contain the supplied absolute directory path. Then update
validateDirectory so its error message is sanitized and excludes user-provided
directory values while preserving the invalid-directory failure behavior.

Source: Coding guidelines

Comment threadagents/release/__tests__/gitOps.test.js Outdated
Comment threadCHANGELOG.md Outdated
Comment on lines +65 to +73
it("should return deterministic lowest-confidence type for generic content", () => {
const issue = {
title: "Something",
body: "TODO",
};
const result = handler.inferType(issue);
expect(result.type).toBeDefined();
expect(result.confidence).toBeLessThan(0.3);
// No patterns match, "todo" is not a keyword → 0 confidence
expect(result.confidence).toBe(0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the actual deterministic selection result.

These tests only assert that a type exists, has non-zero confidence, or belongs to a candidate list. They can pass when the handler returns a type that does not have the highest score.

  • scripts/automation/__tests__/handle-needs-triage.test.js#L65-L73: call inferType twice and compare both selected type and confidence.
  • scripts/automation/__tests__/handle-needs-triage.test.js#L76-L85: derive the maximum entry from result.scores and assert its type and confidence match the returned selection.
  • scripts/automation/__tests__/handle-needs-triage.test.js#L458-L473: assert the processIssue type inference matches its highest score.

As per coding guidelines, “All code changes must include lint fixes, relevant tests, and a short rationale summarising the change”.

📍 Affects 1 file
  • scripts/automation/__tests__/handle-needs-triage.test.js#L65-L73 (this comment)
  • scripts/automation/__tests__/handle-needs-triage.test.js#L76-L85
  • scripts/automation/__tests__/handle-needs-triage.test.js#L458-L473
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/automation/__tests__/handle-needs-triage.test.js` around lines 65 -
73, Strengthen the deterministic selection assertions in
scripts/automation/__tests__/handle-needs-triage.test.js at lines 65-73, 76-85,
and 458-473: call inferType twice at lines 65-73 and compare both selected type
and confidence; at lines 76-85 derive the maximum entry from result.scores and
assert the returned type and confidence match it; at lines 458-473 assert
processIssue’s inferred type and confidence match its highest score. Include any
required lint fixes and a brief rationale summarizing the test changes.

Source: Coding guidelines

Comment on lines +449 to +455
// At threshold 2.0 (impossibly high), effective check: 1.0 >= 2.0 * 0.85 = 1.7
// 1.0 < 1.7 AND no areas → warning
const resultLow = await handler.processIssue(issue, {
dryRun: true,
confidenceThreshold: 2.0,
});
expect(resultLow.status).toBe("warning");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject invalid confidenceThreshold values.

Line 453 treats 2.0 as a supported threshold. The handler accepts it without validation, although inference confidence cannot exceed 1.0. This can silently force every issue into warning.

Validate confidenceThreshold in the range 0 to 1, and change this test to assert the invalid-configuration result.

As per coding guidelines, “Validate all input”.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/automation/__tests__/handle-needs-triage.test.js` around lines 449 -
455, Validate confidenceThreshold at the handler’s input/configuration boundary,
accepting only values from 0 through 1 and returning the established
invalid-configuration result for out-of-range values such as 2.0. Update the
test around handler.processIssue to assert that invalid configuration outcome
instead of a warning, while preserving valid-threshold behavior.

Source: Coding guidelines

@github-actionsgithub-actionsBot removed the type:chore Chore / small hygiene change label Aug 11, 2026
@github-actions

github-actionsBot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🔍 Reviewer Summary for PR #1729

CI Status:success
Files changed: 5
Risk Distribution: 0 critical, 0 high, 2 medium, 3 low

Recommendations

  • Ready to proceed pending human review

@ashleyshaw
ashleyshaw removed the request for review from krugazulAugust 11, 2026 09:26
ashleyshawand others added 4 commits August 11, 2026 11:41
…1716)
**Issue #1715 - Changelog Validation Regex Bugs (3 fixes):**
1. Line 31-40 (changelogValidator.cjs): Fixed em-dash validation that incorrectly flagged all hyphenated words. Now only flags spaced hyphens (` - `) used as pauses, not compound words like "backwards-compatible".
2. Line 174 (keepAChangelogParser.cjs): Replaced Perl syntax `\z` with JavaScript anchor `$` for end-of-string validation.
3. Line 186 (keepAChangelogParser.cjs): Added regex metacharacter escaping for category names to handle special chars like `[Breaking]`.
Added comprehensive test coverage:
- 4 new em-dash validation tests verifying compound words are allowed
- 10 new parser tests covering edge cases (empty content, special chars, boundary conditions)
- Tests verify deterministic scoring behavior
**Issue #1716 - Strengthen Test Assertions:**
Replaced overly-permissive and conditional test assertions with deterministic behavior verification:
- 7 new edge case tests (empty content, whitespace, boundary conditions, deterministic scoring)
- Simplified conditional assertions that tested arbitrary behavior
- Documented confidence threshold multiplier (0.85) in tests
- All tests now verify documented API behavior rather than derived expectations
- Test coverage maintained at 89.06% (handle-needs-triage.js)
All 38 tests passing in handle-needs-triage.test.js
All 32 tests passing across changelog validator and parser tests
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Add unit tests for GitRepository class and utility functions in gitOps.cjs:
- Directory validation tests
- Error sanitization
- Core functionality validation
- Integration tests for git operations
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@ashleyshaw
ashleyshawforce-pushed the fix/changelog-regex-and-test-assertions branch from ecc9590 to 4f15688CompareAugust 11, 2026 09:42
@ashleyshaw
ashleyshaw merged commit f4f5092 into developAug 11, 2026
30 of 32 checks passed
@ashleyshaw
ashleyshaw deleted the fix/changelog-regex-and-test-assertions branch August 11, 2026 09:47
@ashleyshaw
ashleyshaw restored the fix/changelog-regex-and-test-assertions branch August 11, 2026 09:47
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:automationAutomation workflows and agentsarea:documentationDocs & guidesarea:scriptsScripts & toolingarea:testsTest suites & harnesseslang:jsJavaScript/TypeScriptlang:mdMarkdown content/docsmeta:needs-changelogRequires a changelog entry before mergepriority:normalDefault prioritystatus:needs-reviewAwaiting code reviewtype:bugBug or defect

Projects

None yet

Development

Successfully merging this pull request may close these issues.

P2: Test Expectations Misalignment — API Documentation & Coverage P2: Changelog Validation Regex Bugs — Correctness Fixes

1 participant

@ashleyshaw